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CONTEXT OF THIS VOLUME 

This is one in a series of volumes produced by the JTPA EVALUATION DESIGN PROJECT. 



PURPOSE AND PHILOSOPHY 

The purpose of this project has been to develop a set of evaluation tools that are useful to states and loca! service delivery 
areas (SDAs) in judging the way their JTPA programs are being managed and the impact they are having. The intention 
has been to base these analytic and managerial tools on sound program concepts and research methods, and to design 
them such that the information obtained is of practical and direct use in improving JTPA policies and programs at the 
state and local level This kind of information is also expected to make a unique contribution to national training policy 
and Federal oversight of JTPA. J 

It is hoped that these volumes will stimulate and support state and local evaluation efforts in JTPA, and promote more 
consistency than in previous programs with respect to the issues studied and the methods used 10 investigate them \n 
important goal is to encourage the generation of complementary information on program implementation and impact 
that is comparable across states and SDAs. Comprehensive, comparable information is essential to the development of 
a valid and reliable knowledge base for resolving problems and improving programs. It is also required for adjusting na- 
tional training strategies to changing needs and priorities at the state and local level. 

PRODUCTS 

Consistent with this purpose and philosophy, the project has produced a set of materials to assist states and SDAs in 
evaluating their program*. These are to be useful in planning, designing and implementing evaluation activities. As an 
integrated collection, each ;et is developed to support comprehensive evaluations over the JTPA planning cycle. 

The careful tailoring of these materials to state and local users is appropriate. JTPA represents a new employment and 
training policy shaped not oi; y by the experience of managers and the perspectives of employers, but by scientific assessments 
of previous approaches for addressing unemployment, poverty and other barriers to economic security. In this context, 
the value of JTPA programs is also expected to be judged. In fact, the Act's assessment requirements are more explicit 
and sophisticated than those of any employment and training legislation to date. It clearly distinguishes between monitor- 
mg activities, whose purpose is to determine compliance (such as with performance standards) and evaluation activities 
whose purpose is to determine how a program is being managed and implemented, and the kinds of effects it is having 
cn recipients and relevant others. Equally significant, new constituencies are ejected to make these more rigorous 
assessments. States and SDAs now have this important responsibility. It is the first time in the history of employment 
and training programs that the Federal government's evaluation role has been significantly reduced. 

This change affords states and local areas opportunities to influence public policy. It also requires them to assume new 
oversight responsibilities. Program evaluation is expected to become an integral part of the management of organizations 
administering, planning and delivering public training services. This is as it should be. The more information available 
at these levels, where changes in organizations can most readily be made, the more effective the management of JTPA 
programs. This project was undertaken in that context. 

The evaluation tools produced by the project have been developed with a sensitivity to the differing needs, interests 
» r i S r°Tn C A S i? f s , late . and lo i al users - The y h * 7e been packaged into a single comprehensive and integrated set of volumes 

called JTPA Evaluation at the State and Local Level. The set contains planning and evaluation guides and issue papers 

The following volumes are available in the set: 



Volume 
I: Overview 

II: A General Planning Guide 
HI: A Guide for Process Evaluations 
III Supplement: Some Process Issues at the State Level 
IV: A Guide for Gross Impact Evaluations 
V: A Guide for Net Impact Evaluations 
VI; An Implementation Manual for Net Impact Evaluations 
VII: Issues Related to Net Impact Evaluations 

A. Issues in Evaluating Costs and Benefits 

B. The Debrtc Over Experimental vs. Quasi-Experimental Approaches 
VIII: MIS Issues in Evaluating JTPA 

NOTE: Although each of the discrete products listed above is the responsibility of a single author, each seeks to incor- 
porate the results of professional peer review, the many excellent recommendations of the advisory group, and the ideas 
and suggestions of the numerous practitioners interviewed in the process of developing these materials 



Author 

Project Team 
Deborah Feldman 
David Grembowski 
David Grembowski 
Carl Simpson 
Terry Johnson 
Terry Johnson 

Ernst Stromsdorfer 
Ann Blalock 
David Grembowski 



To further qualify these volumes, Volume III is accompanied by a supplement for state users. This is consistent with 
the significant differences between states and SDAs in the kinds of process issues that are most essential to study. The 
volume on net impact evaluations is sufficiently technical, because of the statistical methods involved, that a practical 
manual has been written to accompany it. This guide and manual tend to be more appropriate for states, since relatively 
large sample sizes are required for analysis. However, they are equally useful to larger SDAs and consortia of smaller 
SDAs which may want to jointly study the net impact of their programs. Regional evaluations, for exampi** can be very 
productive in providing management information relevant to regional labor markets. Although ther H * separate issue 
paper on evaluating costs and benefits, this issue is also covered in the gross impact and net impact gw'tfrr h s respect, 
the user benefits from three related but different approaches to this important element of program » •*• ' *ns. Also, 
the user should be aware that the Appendix of Volume II includes A Report on a National/State Sm - >cai JTPA 
Constituencies. This survey was carried out by Bonnie Snedeker, with the assistance of Brian O'SultWai 'de addi- 

tional input from practitioners to the development of the planning and process evaluation guides 

In conclusion, several expectations have directed the development of these volumes: 



THE GUIDES 

The General Planning Guide 

This guide is to assist users in planning, funding and developing an organizational capacity to carry out pr* :\-css, gross 
outcome, and net impact evaluations and to utilize their results. Separate state and local versions are available. 

The Evaluation Guides 
These volumes are to have the following characteristics: 

□The guides are to complement one another. 

•They are to provide information on program management and other characteristics of program implementati >n, which 
can: 

—Describe the way in which administrative, managerial and service delivery policies and practices operate to affect 

outcomes, as a set of interventions separate from the program's services. 
—Pinpoint the source, nature and extent of errors and biases for which adjustments must be made in gross and net 

impact evaluations. 
—Help explain the results of gross and net impact evaluations. 

•They are to provide information on aggregate gross outcomes, and outcomes differentiated by type of service and 
type of recipient, which can: 

—Describe relationships between certain implementation modes and service strategies, and a broad array of client and 

employer outcomes. 
—Help explain the results of net impact evaluations. 

—Suggest the more important outcomes that should be studied in net impact evaluations. 

—Help sort out those aspects of implementation that may be most critical to study in process evaluations. 

•They are to provide information on net impact (the program's return on investment), which can: 
—Closely estimate the effect of the program's services on clients. 

—Suggest which services and client groups are most important to study in broader but less rigorous gross impact studies. 
—Help identify the decision points in program implementation (particularly service delivery) which may be most 
important to study in process evaluations. 

□The guides are to enable the user to carry out comprehensive assessments of JTPA programs. 

•They are to allow the user to acquire several different perspectives on the same program within a particular time period: 
on program implementation, on outcomes for clients and employers and on net impact. 

•They are to permit the user to interrelate these different kinds of information to gain a wider understanding of what 
is happening in a program and why. 

□The guides are to describe approaches and methodologies as consistently as possible, to achieve comparability. 
•They are to define variables and relationships as similarly as possible. 

•They are to define research designs, and methods of data collection and analysis using as similar concepts as possible. 

□The guides are to draw from past research on employment and training programs, as well as seek new approaches and 
methods of specific value in evaluating JTPA at the state and local level. 

•They are to replicate, to the extent possible and feasible, the issues and measures reflected in Federal monitoring and 
evaluation decisions. 

•They are to make selective use of the results of relevant CETA studies, national studies of JTPA, and issue papers 
on JTPA evaluation by national public interest organizations in the employment and training area. 

•They are to rely on the professional literature in applied social research. 



THE ISSUE PAPERS 

• Vo1 "™ V 11 conl ains two issue papers which serve as companion nieces 10 the preceding volumes on net impact evalua- 
tion. The first paper on cost-benefit issues is designed to help users identify, measure and analyze relationships between 
monetary and nonmonetary costs and benefits in determining the program's return on investment. The second paper ex- 
amines the pros and cons of different research strategies associated with the net impact approach. The final volume on 
Mlb issues is to assist users in better understanding how JTPA and other employment and training manasemert informa- 
tion systems can efficiently support the evaluation of program implementation and impact. 

THE SET OF VOLUMES 

The set is integrated, but affords flexible use. The user can utilize the entire set for comprehensive evaluations over 
a two-year planning cycle or longer planning period, or the user can apply the information in each volume independently, 
based on the most pressing evaluation priorities and timeframes and given the extent of resources, during a particular 
fiscal year or biennium. v 



It should be understood that although evaluation products have been developed for JTPA, their basic principles and 
methods can be applied more broadly by states and local areas to evaluate other employment and training programs and 
other social programs. 
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CHAPTER 1. INTRODUCTION 



The Job Training Partnership Act (JTPA) of 1982, which officially 
replaced the Comprehensive Employment and Training Act (CE1A) on 
October 1, 1983, continues CElA's stated objective of providing job 
training to improve the economic well-being of trainees. in 
particular, Section 106 of the Act explicitly states that "job training 
is an investment in human capital," and that the "return on the 
investment is to be measured by the increased employment and earnings 
of participants and the reduction in welfare dependency." Although the 
major objectives of the two programs are similar, the passage of JTPA 
significantly changed the employment and trainir.g system and resulted 
in states and local areas having much greater responsibility for 
program accountability under JTPA than in the past. This has, in turn, 
increased the need for reliable information on program effectiveness at 
the state and local level. 

At the same time that the need for information on program effectiveness 
is increasing, however, the federal government is greatly reducing its 
role in providing such information. As a result, states do not have 
access to reliable research that could assist them in making informed 
and objective judgments concerning the effectiveness of JTPA programs 
and meet their increased oversight responsibilities under the Act. 
Moreover, states that might be interested in conducting their own 
analysis face the difficult problem of choosing a research design from 
among the myriad of approaches that have been used to examine the 
effectiveness of employment and training programs at the national 
level, and then tailoring it to meet their particular analysis needs 
and capabilities. Without a standardized approach to analyze the 
effectiveness of their programs, relatively little program analysis 
research has been conducted at the state level, and the studies that 
have been done have generally used inadequate approaches. To fill this 
gap, in this manual we describe a net impact evaluation model that most 
states could implement to provide much needed information on the 
effectiveness of their JTPA programs. Specifically, this manual serves 
as an implementation guide that describes in step-by-step terms how to 
estimate the net impact of JTPA on earnings and welfare dependency to 
help determine the return on the investment of job training programs 
funded under JTPA. 

Before describing the net impact model, it is important to understand 
the significance of the term "net" and how a "net" impact model differs 
from a "gross" impact model. A gross impact analysis essentially 
compares the post-program labor market experiences of participants with 
their pre-program experiences and attributes all gains to the program. 
Although a carefully designed gross impact analysis can provide useful 
information on the re lative effectiveness of different JTPA program 
activities, because numerous other factors may have changed from the 
pre-program to the post-program period that affect participants' labor 
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market experiences (e.g., Improvement 1n labor market conditions), such 
an analysis 1s not likely to Isolate the true benefits of program 
participation per se, and should not be used to measure the return on 
the job training Investment. On the other hand, a net impact analysis 
compares the labor market experiences of participants with the 
experiences of a comparison group of otherwise similar nonpartldpants . 
The comparison group 1s used to approximate what the labor market 
experiences of participants would have been 1n the post-program period 
had they not participated 1n the program. As such, a net Impact 
analysis only attributes to program participation the Incremental gain 
1n labor market experiences that occurs over and above what would have 
happened had these Individuals not participated 1n the program. This 
1s the appropriate concept for providing Information on the return on 
the Investment of job training programs funded under JTPA. 

The state-level model described 1n this Implementation guide 1s based 
on a detailed review of the employment and training program evaluation 
literature and a thorough examination of alternative approaches to 
conducting net Impact analysis. The literature review, the Issues 
Involved 1n conducting a net Impact analysis, and the advantages and 
disadvantages of alternative approaches are described 1n a companion 
evaluation guide (Volume V 1n this series). The evaluation guide 1s 
Intended to serve as a reference document for those Interested 1n the 
technical details and 1n the reasons why a particular option was 
chosen. This Implementation guide provides very brief justifications 
for why a particular approach was chosen and Instead focuses on how to 
Implement the recommended approach and how to deal with some of the 
more Important problems that are likely to arise. 

In developing a state-level net Impact model of JTPA, we were guided by 
several considerations. First, 1n order for the model to assist states 
1n meeting their new accountability responsibilities, the model must be 
designed to produce valid, estimates of the net Impacts of JlPA programs 
on relevant participant postprogram outcomes. Second, the model must 
be usable and provide meaningful Information that can be understood by 
a nontechnical audience 1n a cost-effective and ti mely manner. 
Finally, to maximize usability, the model must recognize the severe 
resource constraints that states and SDAs face, as well as certain 
other practical considerations . The two most Important practical 
considerations that affected the recommended approach are (1) states 
and local SDAs will not generally be willing to implement an 
experimental design 1n which eligible applicants are randomly assigned 
to treatment-control status, and (2) states and local SDAs will not 
generally be willing to conduct follow-up Interviews with a large 
sample of participants and comparison group members. 

The state-level net Impact model described 1n this Implementation 
manual attempts to meet these objectives while recognizing the Inherent 
tension between them. Because the proposed net Impact model relies 
exclusively on administrative data from several agencies that are 
collected as part of the normal operating process 1n most states, we 
believe that the model 1s usable and that the analysis can be conducted 
1n a timely manner and within the resource constraints that states and 
SDAs face. At the same time, however, such an approach necessarily 
limits the questions that can be addressed and also limits the 
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variables that can be used to adjust for various analytical problems 
that are likely to arise and that could threaten the validity of the 
analysis. To the extent that the model 1s usable and provides valid 
results on Important post-program outcomes (e.g., earnings welfare 
dependency), then an Important by-product, namely c onsistenc y 1n 
application across states, will occur. This will" maximize the 
Information obtained from the analyses and make an Important 
contribution to what 1s known about the effectiveness of employment and 
training programs 1n states that face different environments. 

The remainder of this implementation guide Is organized as follows in 
Chapter 2, we describe the key elements of the conceptual framework for 
a state-level JTPA net Impact model. In particular, we Indicate the 
outcome measures to be examined, the subgroups of trainees for which 
Impacts should be separately measured, the program activities 
(treatments) that should be examined, the types of economic and 
demographic characteristics that may affect these outcomes, and the 
data sources that will be used to measure these elements. 

In Chapter 3, we describe a research design for analyzing the net 
Impact of JTPA. The research design Includes a description of how to 
select a reliable comparison group and also offers guidance on how to 
select a representative sample of JTPA participants and sufficient 
numbers of participants and comparison group members to provide valid 
net Impact results. 

In Chapter 4, we discuss the sters Involved 1n obtaining and processing 
the various data sources for the state-level net Impact model In 
particular, we describe the elements of a data collection and 
processing cycle, Indicating potential problems that may arise 1n the 
process, describe how the various data sources should be organized the 
types of data cleaning that should be performed, and briefly describe 
key features of the analysis files that should be constructed. 

In Chapter 5, we present a data analysis plan to estimate the net 
impacts of JTPA programs on participants' post-program outcomes. This 
section begins with a description of methods for determining the 
quality of the comparison groups selected. We then describe a 
recommended approach to estimating the average net impacts of J1PA, and 
the net impacts for major subgroups, on the key participant 
post-program outcomes Indicated in the legislation. We also discuss 
potential threats to the validity of the analysis and indicate possible 
approaches for adjusting for such problems. 

In Chapter 6, we describe how to conduct a cost analysis to determine 
the costs of JTPA services. We also Indicate how the cost information 
should be combined with the net impact estimates to provide Information 
on the return on the Investment of job training programs funded under 

Finally, in Chapter 7, we briefly indicate how the net impact analysis 
complements the other models being developed and how information from 
the other models may be useful in conducting the net impact analysis 
and in Interpreting the net. Impact results. 
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CHAPTER 2. CONCEPTUAL FRAMEWORK 



An Important element In the design of a state-level JTPA net Impact 
model Is the conceptual framework. The conceptual framework Identifies 
the key research questions to be addressed, the outcomes to be 
examined, the participant groups and program activities (treatments) to 
be Included, and the specific definitions of the outcomes, treatments, 
and variables that affect the relationship among treatments and 
outcomes. Because the net Impact model Is based exclusively on 
available administrative data, the conceptual framework Is In large 
part data-determined J In this chapter, we describe a conceptual 
framework for conducting a state-level JTPA net impact analysis based 
entirely on available state administrative data sources. 



GENERAL RESEARCH QUESTIONS 

Although an employment and training program such as JTPA may affect 
different groups (e.g., participants, employers, government, or society 
as a whole) In different ways, the primary goal of the state-level JTPA 
net Impact model Is to determine the extent to which JTPA programs 
Improve the labor market experiences of participants relative to what 
their experiences would have been In the absence of the program, that 
1s, relative to a comparison group of otherwise similar 
nonpartldpants. The average net Impacts of JTPA programs an 
participants' post-program labor market experiences will provide 
policy-makers with an Indication of the overall gains due to these 
programs that will more than meet state's Increased accountability 
responsibilities under JTPA. Moreover, although It Is Important to 
know whether the mix of JTPA programs Is effective on average, ior 
policy purposes It Is perhaps more Important to probe beneath the 
average Impacts to Identify the relative effects on different 
subgroups. For example, to Improve future targeting practices It Is 
Important to know whether certain participant groups benefit more from 
JTPA than other subgroups, and whether the net Impact of JTPA differs 
among program activities (treatments) or length of participation. In 
addition, 1t Is Important to determine the extent to which net Impacts 
vary by local environmental conditions (e.g., unemployment rate). 
Because previous national studies* did not have the necessary data to 
address this latter Issue, It may be possible for the state-level model 
to make a unique contribution to what Is known about the effectiveness 



Even though the conceptual framework 1s constrained by the 
features of available state data bases, 1t Is Important to note that 
virtually all previous national studies of the net Impacts of 
employment and training programs share several of the limitations of 
the state-level model. The net Impact evaluation guide (Volume V) 
discusses this Issue In detail. 



of employment and training programs that operate in different 
environmental conditions. 

The genera] objectives of the state-level net impact model can be 
summarized by a series of key research questions to be addressed: 

• What is the overall net Impact of JTPA programs on participants' 
post-program labor market experiences? 

• How do the net impacts change over time? 

• For which program activities (treatments) are the net impacts 
the largest? That is, which program activities result in the 
largest net benefits to participants? 

• For which groups of participants are the net impacts the 
largest? That is, which subgroups gain the most from 
participating in JTPA? 

• Do individuals who remain in JTPA longer experience greater net 
gains in labor market outcomes? 

• How does the net impact of JTPA vary by local program 
environmental conditions? 



These are the general research questions that the state-level JTPA net 
impact model will examine. In Exhibit 1 we survey some the key 
elements of the conceptual framework of the net impact model that help 
to make these questions more specific by indicating the outcomes that 
will be used to measure participants 1 labor market experiences, by 
indicating the participant subgroups to be examined, by identifying the 
types of program activities that will be examined, by identifying the 
local program environmental conditions of interest, and by Indicating 
the various data sources that will be required. Below we discuss each 
of these key elements of the conceptual framework. 



PARTICIPANT OUTCOMES 

As indicated In Exhibit 1, the state-level JTPA net impact model will 
focus on the following post-program participant outcome measures: 

• Quarterly, semi-annual, and annual earnings and employment 
status based on UI Wage Records; and 

• Quarterly, semi-annual, and annual AFDC grants and AFDC 
participation status based on Public Assistance (PA) Grants 
Records . 



Because UI Wage Records are maintained on a calendar quarter basis, the 
earnings measures for longer periods can be created directly as sums of 
quarterly earnings values. The AFDC grants measures can be calculated 
as the sum of monthly grant payments. The employment status and the 
AFDC participation status outcome measures will be defined as 
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Exhibit 1 



KEY ELEMENTS OF THE CONCEPTUAL FRAMEWORK 
FOR THE JTPA NET IMPACT MODEL 

Outcome Measures 

• Quarterly, semi-annual, and annual earnings and employment 
status based on UI Wage Records 

• Quarterly, semi-annual, and annual AFOC grants and AFOC 
participation status based on PA records 

Participant Subgroups 

• Adult men and women (to the extent possible, net impacts will 
be disaggregated by age, race/ethnicity, education, and for 
women, by welfare status) 

Program Activities 

• Classroom Training (perhaps separately for remedial education 
and basic skills versus specific occupational skills training) 

• On-the-job Training 

• Job Search Assistance 

• Other 

Labor Market Conditions 

• Unemployment Rate 

• Urban/Rural Location 
Data Sources 

• JTPA MIS (Participant characteristics, program activities, 
placement experiences) 

• PA Grants Records (AFOC grants, whether received AFOC) 

• UI Wage Records (Quarterly earnings, whether employed) 

• UI Benefit History (Pre-program UI payments received, whether 
received UI) 

• Local Labor Market Information (Unemployment rate) 
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categorical or dummy variables- e.g., 1 1f employed during the period, 
0 otherw1se--1n each of the different post-program periods of Interest. 

These outcomes are consistent with the major objectives of the 
legislation and, for the most part, greatly exceed the outcomes 
examined 1n previous national studies of the net Impacts of employment 
and training programs. They also capture the range of short-term and 
relatively long-term Impacts that could be observed within 
approximately a two-year program analysis cycle. Although potentially 
valuable Information on the mechanisms through which employment and 
training programs Increase earnings could be provided by examining 
other outcome measures such as employment Intensity (e.g., hours worked 
per week, weeks worked per year), hourly wage rates, job retention, or 
the additional outcomes listed 1n the legislation for youths, such 
outcome measures require the collection of survey data from both 
participants and comparison group members. This data collection effort 
would greatly Increase the resources required to Implement the net 
Impact model, and thus reduce Its usability. However, states with 
additional resources, or particular Interest 1n some of these other 
dimensions of labor market experiences, are encouraged to collect the 
necessary survey data and follow the research design and analysis plans 
described 1n subsequent sections to estimate the net Impact of JTPA on 
these additional outcomes. 

As discussed 1n the evaluation guide, the administrative data sources 
recommended for constructing the key participant outcome measures have 
several advantages over other potential data sources. For example, 
relative to SSN earnings records that have been used 1n most previous 
national studies, UI Wage Records are not "top-coded" at the SSN 
taxable maximum; they are available on a much more timely basis; and 
because they are available quarterly, one has more flexibility than 1s 
possible with SSN data that are available only on an annual basis. 
Moreover, although survey data allow for much more flexibility-- 
including a greater range of outcome measures and personal characteris- 
tics—in Wage Records are not subject f o Interviewer biases; nor to 
problems that arise from some respondents reporting net (after-tax) 
earnings and others reporting gross (bef ore-tax) earnings; and they are 
not affected by response- rate problems that can plague survey data 
collection efforts . Monthly AFDC grants from administrative records 
also have numerous advantages relative to data obtained through 
surveys, Including timeliness of data availability, and not being 
affected by respondent reporting errors, Interviewer biases or 
response-rate problems. As a result of these advantages and the fact 
that they are quite Inexpensive to obtain, the net Impact model has 
been designed to rely on these administrative data systems. 

Although these administrative data sources have several advantages, 1t 
1s also Important to recognize their limitations. First, and foremost, 
the 12 states that are not wage- reporting will generally not be able to 
use the net Impact model to examine earnings Impacts at this time. 2 



* The 12 wage- request states are Hawaii, Massachusetts, Michigan, 
Minnesota, Nebraska, New Jersey, New York, Ohio, Rhode Island, Utah, 
Vermont, and Wisconsin. The Deficit Reduction Act of 1984, however, 
requires that all states effectively become wage-reporting by 1988. 
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The potential exceptions are states like New York that, although 1t 1s 
a wage-request state, comparable earnings records are maintained by the 
Department of Revenue and could be used for the analysis, provided the 
necessary Interagency agreements could be made (e.g., data conflden- 

Second, states that do not have automated AFOC grants records available 
at the state level will not be able to easily use the net Impact model 
to examine JTPA impacts on reducing welfare dependency. Third, because 
the model relies heavily on the availability of historical earnings and 
grants records to control for potential differences between 
participants and comparison group members in the pre-program period 
(see Chapter 5), states must be able to directly access (or retrieve 
from archives) at least two to three years of UI wage records and AFDC 
grants records for a given person at any one time. Although most UI 
Wage Records systems are quite similar across wage- reporting states and 
generally contain at least seven quarters of data at any time, there 1s 
considerable variation in stat* and local welfare administration and 
record-keeping practices, as well as differences in the degree of data 
automation and retrieval capabilities, that will present obstacles to 
Implementing this component of the net Impact model in certain states. 
Fourth, UI Wage Records do not generally Include employees of federal, 
state, or local governments, self-employed individuals, or employees in 
certain other occupations. 3 

Finally, because the system is state-based, 1t is Impossible to 
distinguish Individuals who work across the border 1n a different state 
from individuals who do not work in covered employment. Thus, unless 
Interstate agreements can be worked out to access UI Wage Records, one 
should be very careful in estimating the net impact of JTPA on earnings 
for large SDAs located near state borders. However, as we indicate 1n 
Chapter 5, the deficiencies of UI Wage Records (i.e., coverage and 
border problems) should not 1n general result in biased estimates of 
net program Impacts. 



PARTICIPANT SUBGROUPS 

The next conceptual framework issue concerns the participant groups for 
which the outcome measures described above will be examined to 
determine the net impact of J1PA programs. As indicated 1n Exhibit 1, 
the state-level net Impact model has been designed for adults only. 
This 1s 1n part because the outcome measures listed above (earnings, 
employment, and AFDC dependency) are not appropriate for youths 
(particularly 1n-school youths), and no existing data sets include 



For example, in the State of Washington, UI Wage Records also do 
not Include earnings for the following types of employees: certain 
corporate officers, church employees, individuals paid exclusively on 
commission, domestics who work for employers who pay less than $1,000 
per quarter, railroad employees, employees of small agricultural firms, 
casual laborers, and certain barbers/cosmetologists. As a result of 
these coverage gaps, approximately 80 percent of all state wages are 
included in the UI Wage Records data base. 
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information on more appropriate outcome measures (e.g., schooling 
attainment, employment competencies) for both participants and 
comparison group members* Moreover, extensive evidence presented 1n 
the evaluation guide Indicated the difficulty researchers have had 1n 
developing a reliable matched comparison group for youths wtfo have 
limited pre-program earnings histories. Because of these problems, the 
net Impact model has been designed only for adults, and states 
Interested 1n examining youth Issues should consult the implementation 
guide for the gross *mpact model or consider other more costly 
approaches for conducting a net Impact analysis. 4 

A second issue concerns whether separate net Impact models need to be 
developed for specific adult subgroups. As described 1n the evaluation 
guide, most previous studies estimated separate net Impact models for 
men and women; some of these studies also estimated separate models for 
youths and adults, and others estimated separate models by race or 
ethnicity group. We recommend that separate net Impact models be 
developed for adult men and women because of extensive evidence 
Indicating that the relationship between earnings and other demographic 
characteristics 1s very different for these two groups. However, we do 
not believe 1t 1s necessary to estimate entirely separate net Impact 
models by race or for other participant subgroups. 

Even though entirely separate models will not be .estimated for other 
participant subgroups, as we describe in Chapter 5, 1t is possible to 
design the analysis to examine whether the net impact of JTPA differs 
among other adult male and female subgroups of interest. For example, 
based on the previous literature, 1t will be important to Investigate 
whether the Impact of JTPA varies by the following participant 
characteristics : 

• Age (less than or equal to 35 as compared to over 35); 

• Ethnicity (whites as compared to blacks and Klspanics); 

• Educational level (at least a high school graduate as compared 
to nongraduates) ; and, 

• Welfare status for adult women (welfare recipients as compared 
to nonreciplents) . 

To the extent sample sizes are k adequate and states have the necessary 



4 For example, states that are very Interested in developing net 
impact estimates for JTPA Title II-A youth prcgrams might consider 
implementing an experimental design, or alternatively, conducting 
(relatively expensive) Interviews of participants and comparison group 
members to collect the detailed pre-program and post-program employment 
and schooling data necessary for reliable analysis, and then follow the 
research design and analysis plans described later in this 
implementation guide. States interested 1n such approaches, however, 
should first consult employment and training researchers who are 
knowledgeable 1n experimental design issues and questionnaire 
development to avoid the pitfalls that have plagued previous studies. 
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staff/computer resources, as Indicated 1n Exhibit 1 we recommend that 
net Impact estimates be derived for these subgroups of adult men and 
women 1n order to replicate the research questions examined In previous 
studies and to provide valuable Information on targeting issues 



PROGRAM ACTIVITIES 

Another Important element of the conceptual framework 1s the 
determination of the key program activities (treatments) ;o be examined 
and the development of consistent definitions of the treatment 
variables. Section 204 of the Job Training Partnership Act authorizes 
the expenditure of Title I I-A funds for over 20 types of employment and 
training program activities. Although the 11st of potential program 
activities 1s quite extensive and seems to allow a virtually unlimited 
array of potential services, the major program activities provided 
under JTPA Include classroom training (CT), on-the-job training '0J1) 
and job search assistance (JSA). In fact, nearly 90% of adult FY 1984 
Title II-A enrollees participated 1n one of these programs. Although 
work experience was used extensively under CETA, only 3% of adult 
participants 1n FY 1984 were assigned to work experience programs. 
Thus, although 1t 1s Important to Include 1n the analysis participants 
who are assigned to all types of activities 1n order to get an unbiased 
estimate of JTPA overall, as Indicated 1n Exhibit 1 1t 1s likely to be 
most useful to examine the separate effects of CT, OJT, and JSA. 5 A 
brief description of each of these program activities 1s provided below. 

• Classroom training Involves basic or remedial educational 

training, or occupational skills training to ensure that 
Individuals acquire the ability and knowledge necessary to 
perform a specific job for which there 1s a demand. Such 
programs are usually provided 1n a classroom or Institutional 
setting. 

• On-the-job training emphasises the development of occupational 
skills 1n an actual work setting, normally 1n the private 
sector. The programs are designed for participants who have 
been first hired by the employer, and the training occurs while 
the participant 1s engaged 1n productive work that provides 
knowledge or skills essential to the full and adequate 
performance of the job. 

• Job^ — search assistance Includes any training activity that 

focuses on the development or enhancement of employment- seeking 
skills. This service 1s provided to participants who need 
practical experience 1n Identifying and Initiating contact and 
Interviewing with prospective employers. It 1s usually 
conducted 1n a structured setting and can Include approaches 



If, however, work experience or some other program activity 1s 
used extensively 1n a particular state so that the sample sizes are 
sufficient to support precise net Impact estimates, 1t 1s possible to 
follow the other procedures outlined 1n the Implementation guide to 
estimate the net Impacts for this activity. 
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such as job-finding clubs or Instructions for self-directed job 
search methods. 

In creating specific treatment variables to represent these program 
activities, several potential complications arise. First, although the 
broad definitions of the major program activities are generally 
accepted, there 1s likely to be considerable variation across SDAs 1n 
the contents of specific program activities such as length of 
assignment, occupation of training, hours per day. This makes 1t 
particularly difficult to create meaningful variables that represent a 
homogeneous treatment. At the same time, however, 1t 1s necessary to 
do a considerable amount of aggregation of activities that are 
generally similar but perhaps far from Identical, because 1t 1s simply 
not possible to reliably estimate the net effects of the virtually 
unlimited number of program activities. Second, not only are there 
differences In the degree of treatment within program activity, there 
are likely to be large differences across SDAs 1n the nature of 
programs provided such that work experience programs in a particular 
SDA may be more similar to OJT programs in another SDA. Finally, the 
way in which the actual training activities provided are recorded in 
the program MIS can result 1n additional complications. For example, 
due to the lack of uniform national reporting requirements, some SDAs 
record participation 1n a job search workshop as job search assistance, 
while others record It as classroom training because the sessions are 
conducted 1n a classroom setting. Such differences in the content and 
recording of program activities across SDAs emphasize the importance of 
conducting a process analysis concurrently with the not impact analysis 
in order to develop meaningful and consistent measures of program 
activities. 

As indicated above, the ways in which the treatment variables are 
defined will in large part be determined by the structure and content 
of the MIS. In addition, they will depend on the specific research 
questions of interest, and the sample sizes of individuals who 
participate in the given program activity. For example, to ensure that 
the treatment variables are as homogeneous as possible, it may be 
desirable to separate classroom training activities that focus on 
remedial education and basic skills from classroom training activities 
that provide specific occupational skills training. At the same time, 
however, if the number of Individuals participating 1n each of these 
programs is too small to produce statistically reliable net Impact 
estimates for the separate activities, it may be necessary to collapse 
these two variables into one that represents classroom training 
programs 1n general . 

Thus, although the specific definitions of the treatment variables will 
depend on several factors, the following variables should be created 
for participants for potential Inclusion 1n some models to examine 
Impacts by program activity and other characteristics of the treatment: 

• Participant dummy variable (1 1f JTPA participant; D otherwise); 

• Classroom training dummy variable (1 if CI participant; D 
otherwise) ; 



- Remedial education and basic skills dummy variable (1 1f CI 
program In remedial education and basic skills; 0 otherwise)- 

- Specific occupational skills training dummy variable (1 If 
CT program 1n specific occupational skills; 0 otherwise); 

On-the-job training dummy variable (1 if OJT participant- 0 
otherwise); ' 

Job search assistance aummy variable (Includes all employment/ 
placement-related activities) (1 1f jsa participant- 0 
otherwise); 

Other activity dummy variable (1 If not a CT, OJT or JSA 
participant; 0 otherwise); 

Occupation of training dummy variables (1 if i n specific 
one-digit 001 code; 0 otherwise); 

Length of program participation In weeks; and, 
Number of hours of training per day. 



Such treatment variables would enable one to replicate all of the 
questions examined In previous national studies of employment and 
training programs, as well as several additional questions of Interest 



PROGRAM ENVIRONMENTAL CONDITIONS 

The final element of the conceptual framework concerns the program 
environmental conditions to be Included In the net Impact model By 
program environmental conditions, we are primarily referring to 
characteristics of the labor markets within which the program operates, 
although major SOA characteristics could also be considered As 
Indicated earlier, because of data limitations, previous national 
studies were unable to Include any program environmental 
characteristics In their models. As such, little Is known about how 
the net Impact of employment and training programs vary by program 
environmental conditions. At the same time, however, because of the 
nature of local program environmental conditions (I.e., there Is no 
wlthln-SOA variation on these conditions), It Is Important to recognize 
that It will only be possible to obtain reasonably precise estimates of 
a few key conditions, and only In states that have a large number of 
SDAs, and that exhibit considerable variation In the conditions across 
SOAs. 

As Indicated In Exhibit 1, we believe that it Is most Important to 
control for differences In local unemployment rates and whether the 
Individuals are located In an urban or rural urea. These are the 
factors that the evaluation guide Indicated are most likely to affect 
the employment and earnings experiences of adult men and women. 

Ihe unemployment rate can be obtained from the Local Area Unemployment 
Statistics (LAUS), published by the Bureau of Labor Statistics. This 
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Information 1s available monthly at the state and county level and for 
over l f 000 cities with population of at least 25,000. Aggregate 
measures of the unemployment rate corresponding to the quarterly, 
semi -annual , and annual outcome periods of interest car be calculated 
as weighted averages of the monthly values, where the weights are the 
ratios of the monthly values of the number of Individuals 1n the labor 
force divided by the sum of these monthly values over the outcome 
period of Interest. 

In constructing the unemployment rate variable for an SOA, 1t 1s also 
Important to recognize that the monthly values will not generally be 
available for the precise area of Interest. Depending on the size of 
the SOA, the area 1t serves may be either larger or smaller than the 
county or the city for which the Information 1s available. In cases 
where the SOA serves multiple counties, one should calculate the 
appropriate labor market variables by aggregating over the counties 
served by the SOA. For example, to calculate the unemployment rate one 
would simply sum the number of Individuals unemployed 1n the various 
counties served by the SOA and divide by the total number of 
Individuals 1n the labor force in those counties. In cases where the 
SOA serves only part of a given county, and where no value 1s available 
for a smaller geographical area (I.e., a city), one 1s generally 
constrained to use the county value. 

It may also be possible to provide some Information on how the net 
Impact of JTPA varies by different SOA service delivery strategies. 
The service delivery strategies to be examined should be based on their 
policy Importance to the particular state doing the analysis. 
Moreover, to ensure that the strategies of Interest are distinct and 
quantifiable, and that there 1s sufficient variation among SOAs to 
support the analysis, 1t 1s Important that a process analysis be 
conducted. Thus, 1f states with a large number of SOAs (roughly 30 or 
more) are Interested 1n obtaining some Information on how the net 
Impact of JTPA varies by a key service delivery strategy, they should 
first ensure that there are significant differences 1n this strategy 
across SOAs. Provided 1t 1s possible to quantify these differences, 
one could then use the variable created to determine how the net Impact 
of JTPA varies across SOAs that d1 f fer 1n this strategy using the 
approach described 1n Chapter 5. In states with relatively few SOAs, 
1t 1s very unl 1kely that such an analysis would provide reasonably 
precise estimates of the differential effects of the strategy of 
Interest. 
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CHAPTER 3, RESEARCH DESIGN 



To provide valid estimates of the net Impacts of JTPA programs on the 
earnings and AFDC dependency of adult men and women, a research design 
must be developed that contains several elements. First, a sample of 
JTPA participants must be chosen so that the results can be generalized 
to the state level, and so that the necessary data can be collected 
efficiently, with minimum burden to state and SDA data processing 
staff. Second, a valid comparison group must be chosen ->o that the 
Impact of JTPA can be distinguished from the Impacts of other factors 
that also affect earnings and welfare dependency. Third, the sizes of 
the participant and comparison samples must be determined so that 
program Impacts can be measured with precision. Fourth, the data must 
be obtained and processed, and. analysis files must be developed. 
Finally, an overall estimation strategy must be developed, and analysis 
models must be specified that can provide valid estimates of the net 
Impacts of JTPA programs on the post- program outcomes of participants. 
In this chapter, we describe the first three elements of the research 
design for conducting a state-level JTPA net Impact analysis; data 
collection and processing Issues are described In Chapter 4, and the 
overall estimation strategy and the specific net Impact models to be 
estimated are described 1n Chapter 5. 

An overview of the major components of the first three elements of the 
research design Is provided In Exhibit 2. Below we discuss each of 
these elements 1n more detail. 



PARTICIPANT GROUP 

An Important component of the research design Is the development of the 
participant group. The two major Issues In selecting the participant 
group concern: (1) the Individuals to be Included In the sample frame 
and (2) the procedure to be used to select participants from the sample 
frame for Inclusion In the analysis. A recommended approach to each of 
these Issues 1s described below. 



Sample Frame 

The choice of the sample frame from which the JTPA participant group 
will be drawn 1s an Important determinant of the external validity of 
the net Impact analysis. I.e., the degree to which the findings can be 
generalized. The sample frame should be representative of all J1PA 
participants so that the analysis results can be generalized to the 
state level, rather than just to particular subpopulatlons of 
participants or Individual SDAs. Based on a detailed review of the 
advantages and disadvantages of alternative procedures described In the 
evaluation guide, the state-level net Impact model has been designed to 
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Exhibit 2 



KEY ELEMENTS OF THE RESEARCH DESIGN 
FOR THE J TP A NET IMPACT MODEL 



Participant Group 

• Comprised of samples of adult men and women who enroll 1n JTPA 
1n each quarter of a given program year. 

• Individuals will be excluded from the sample frame 1f they are 
not between 22 to 64 years of age. Individuals will 
subsequently be excluded from the analysis samples 1f they 
have missing data on key JTPA services received (e.g., program 
activity, length of participation). 

• Quarterly samples of JTPA participants will be selected 
randomly f rom the groups of adult men and women enrol lees that 
are sncluded 1n the sample frame to ensure that the sample 1s 
representative of JTPA participants 1n the state. 

Comparison Group 

• Comprised of samples of adult men and women who are new ES 
registrants 1n offices 1n the areas served by the SDAs 1n each, 
quarter of a given program year. 

• Individuals will be excluded from the: comparison sample frame 
If they are not between 22 to 64 years of age; 1f they are not 
economically disadvantaged; or 1f they participate 1n JTPA or 
receive significant ES services. 

• Quarterly samples of comparison group members will be selected 
from the sample frame of new ES registrants using a stratified 
random process to ensure that ES registrants and JTPA 
participants are similar on a few key characteristics (e.g., 
welfare recipiency, UI recipiency). 

Sample Size 

• Because the marginal cost of Increasing sample size 1s very 
small, states are encouraged to Include 1n the analysis as 
many participants and comparison group members as their staffs 
and computer resources can handle. 

• As a guideline, a total analysis sample of 12,000 
cases--d1v1ded equally between adult men and women, and 
between participants and comparison group members (I.e., 3,000 
each)--should be adequate to meet most state's analysis needs. 
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Include in the participant sample frame all JTPA enrollees during a 
given time Interval, as Indicated 1n Exhibit 2. Specifically, the 
participant group will be comprised of samples of adult men and women 
who enroll 1n JTPA 1n each calendar quarter of a given program year. 

A sample frame of all J1PA enrollees from each quarter 1n a given 
program year has numerous advantages. First, it yields a 
representative sample of JTPA participants 1n which neither short-term 
nor long-term participants are oversampled, and which is not sensitive 
to seasonal differences in the characteristics of participants or 
program activities. Thus, 1t should be possible to generalize the 
results to the state level. Second, because the time period for 
selecting each participant cohort within the program year is not too 
long (i.e., three months), 1t should be possible to select quarterly 
samples of comparison group members that closely match participants on 
the timing of the pre-program decline 1n earnings. As Indicated 1n the 
evaluation guide, this 1s particularly Important for ensuring valid net 
impact results. Third, using an enrollee-based sample maximizes the 
amount of pre-program earnings and AFDC data available for the model, 
which is particularly important for states that do not retain extensive 
historical data. 

On the other hand, 1t must be recognized that an enrollee-based sample 
frame has some disadvantages. For example, 1n order to avoid excluding 
long-term stayers from the analysis, an enrollee sample frame causes a 
delay 1n analysis findings relative to a terminee-based sample. In 
addition, because a given group of enrollees may terminate in several 
different quarters, with such a design it 1s more difficult to estimate 
earnings impacts that correspond to specific time periods after program 
termination (e.g., three months following the quarter after 
termination). As described in the evaluation guide, however, 
alternative sample frames suffer from other problems that were 
considered to be more severe, which led us to the decision to use an 
enrollee-based sample. 

Although a participant group comprised of adult men and women who 
enroll 1n JTPA in each of the four quarters of a given program year 
yields a representative sample that can provide valid results, it has 
Implications for the timing of project results and the length of the 
post-program observation period within an approximate two-year program 
analysis cycle. Specifically, given the variation in length of stay, 
the delays Involved in obtaining the outcome measures from agency 
records (approximately three months) and the time required to build the 
data files, conduct the analysis, and prepare written reports 
(approximately six months), with an enrollee-based sample of 
participants one can obtain net Impact estimates for the period 
one year following tho calendar quarter after termination for the 
longest stayers only for the first quarter cohort, and only a 
three-month net impact estimate can be obtained for all four quarterly 
cohorts 1n approximately a two to two and one-half year cycle. Of 
course, by obtaining additional post-program UI and AFDC records for 
sample members, one could estimate longer-term Impacts, although the 
analysis period would have to be extended even further. 
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sample Exclusions 



Once the general sample frame 1s chosen, one must then determine 
whether certain types of Individuals should be excluded. Although any 
such exclusions necessarily reduce the representativeness of the 
participant sample, there may be valid reasons for excluding some 
participants. For example, 1t may be desirable to exclude cases 
because they lack data on critical Items or because they are clearly 
Inappropriate to Include 1n the analysis. Below we offer our 
recommendations on participant sample exclusion Issues for the net 
Impact model . 

As described 1n the evaluation guide, most previous studies have 
Incorporated some restrictions on participant age. Although there 1s 
no universal agreement on the specific age range to use, very young 
participants (e.g., under age 16) have been excluded because earnings 
1s not an appropriate outcome measure for those likely to return to 
school, and very old participants (e.g., those age 65 and older) have 
been excluded because participation 1n employment and training programs 
among Individuals eligible for retirement 1s very rare, and 1t 1s 
unlikely that a proper comparison group could be Identified. Because 
the state-level net Impact model 1s designed to focus on adults only, 
the participant sample will be restricted to Individuals of at least 
age 22. Because of the difficulty of obtaining a valid comparison 
group for older participants, we also recommend that any Individuals 
age 65 and older be excluded from the participant sample frame. 

Individuals should also be excluded from the sample frame 1f they have 
missing data on key variables. For the most part, we do not expect 
that there will be severe missing data problems with the agency records 
to be used, and most problems can be d alt with 1n straightforward ways 
1n developing the analysis variables, as described 1n Chapter 4. 6 A 
more difficult problem arises when Information 1s missing on the 
treatment provided by JTPA. For example, one cannot estimate the net 
Impact by program activity or by length of stay for Individuals with 
missing Information on program activity or for those who have 
Incomplete data on the start and end dates of their JTPA 
participation. Although we expect there will generally be very few 
problems concerning the omission of program start and end dates (1n 
part because length of stay Information 1s necessary for adjusting 
certain performance standards for JTPA Title II-A programs), because 
there are no reporting requirements concerning program activity, 1t 1s 
likely that some cases will contain missing (or unusable) program 
activity Information. Moreover, because the quarterly enrollee samples 
will be selected on an ongoing basis, 1t will not be possible to know 
for sure how many cases must be excluded for missing data problems 
until after the JTPA MIS data are obtained and merged for the sample 
selected. As such, we recommend .that a slightly expanded participant 
sample Initially be selected, and that Individuals who are subsequently 



0 The limited amount of missing data 1s 1n part a reflection of the 
procedures used by many agencies to assign "default" values when data 
are missing. Such procedures lead to measurement error, which can also 
Introduce analytical complications as discussed 1n Chapter 5. 
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As indicated 1n Exhibit 2, we recommend that the quarterly samples of 
JTPA participants be selected on a random basis from the groups of 
adult men and women enrollees Included 1n the sample frame. Although 
adult men and women are served 1n approximately equal numbers by JTPA 
overall (I.e., 52.8% of all Title II-A adult participants were women 1n 
PY 84), because this varies considerably across SDAs, we believe 1t 
would be prudent to first stratify the participant sample by sex before 
the analysis samples are selected, or else there may be Insufficient 
numbers of either men or women for analysis purposes. 8 Choosing a 
random sample from the sampling frames, separately by sex, also has the 
major advantage of providing representative samples of JTPA adult men 
and women participants, so that the results can be generalized to the 
state level. Moreover, estimates of the net Impacts of JTPA and of the 
differential Impacts by program activity can be obtained 1n a 
straightforward manner (I.e., no weighting 1s necessary). In addition, 
by selecting participants randomly, and not based on the particular 
program activity they are enrolled 1n, an analysis of the types of 
Individuals who are assigned to various JTPA program activities (I.e., 
targeting) 1s also straightforward. 

In addition to stratifying the sample by sex, states that want to focus 
their efforts on specific subgroups of adult men or women (e.g., female 
welfare recipients, male or female high school dropouts) may also want 
to consider stratifying the participant sample and oversampling the 
subgroups of Interest. In general, stratification 1s desirable only 
when the research questions of interest relate to subgroups that occur 
rarely, or that occur so frequently that their nonoccurrence is rare. 
Depending on the specific research questions of Interest, one could 
consider stratifying on the basis of participant characteristics or by 
program activity. For example, because of the wide variation across 
states and SDAs in the use of work experience programs, states that are 
interested in examining the net Impact of work experience programs 
would likely need to stratify and oversample participants of such 
programs. Moreover, because job search assistance generally 
constitutes a less Intensive treatment that 1s likely to have a smaller 
average net Impact, a much larger sample of participants in JSA would 
be needed to precisely measure the lower expected effect. Thus, for 
states that are very interested 1n precisely measuring the marginal 
benefits from JSA participation, such participants would have to be 
oversampled. States that may be interested 1n stratifying the 
participant sample and oversampling certain groups should consult a 
statistician or sampling expert to better understand the advantages and 
disadvantages of such an approach and the specific steps to be followed 
1n drawing the sample and conducting the analysis. 



COMPARISON GROUP 

To estimate the net impact of JTPA on participants 1 post-program 
outcomes , a method 1 s needed to gauge what would have happened to 



For example, in some SDAs women comprised as little as 25% of 
adult JTPA termlnees in PY 84, while in other SDAs women were over 80% 
of all adult termlnees in Title II-A programs in PY 84. 
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participants had they not participated 1n JTPA. A standard approach 
for determining the net Impact of a program 1s to compare the 
experiences of persons Influenced by the program (JTPA participants) 
with the experiences of persons who are not Influenced by the program 
(the comparison group). The comparison group 1s used to estimate what 
the experiences of the participants would have been 1n the post-program 
period had they not participated 1n the program. To ensure that the 
differences between the experiences of the participant and comparison 
groups can be attributed to the program, the comparison group must have 
characteristics similar to JTPA participants, particularly 1n terms of 
the characteristics related to JTPA eligibility, and the data available 
must be comparably measured for the two groups. 9 In addition, 1t 1s 
Important that comparison group members not have received significant 
employment and training program assistance, and that a method be 
developed to verify that Individuals 1n the comparison sample 1n fact 
did not receive JTPA services. Following we describe the recommended 
comparison group for the net impact model. 

As Indicated 1n Exhibit ;?, based on a detailed review of several 
alternative data bases, we recommend that the comparison group be 
comprised of new ES registrants 1n offices 1n the areas served by the 
SDAs. lu ES registrants have several advantages as a comparison 



y According to the JTPA legislation, to be eligible for Title II- A 
programs, adults must be 22 years of age or older and be economically 
disadvantaged. The legislation defines economically disadvantaged to 
mean an Individual who: (1) 1s a member of a family that 1n the six 
months prior to application received a total Income of less than the 
0MB poverty level or less than 70% of the lower living standard Income 
level, whichever 1s greater, given the person's family size; (2) 1s a 
member of a family that receives federal, state, or local cash welfare 
payments; (3) 1s receiving food stamps; (4) 1s a foster child for whom 
state or local support payments are made; or (5) 1s a handicapped 
Individual who 1s economically disadvantaged but whose family 1s not, 
as permitted by the Secretary of Labor. The Act requires that at least 
90% of Title II-A participants be economically disadvantaged and allows 
up to 10% of the participants to be Individuals who are not 
economically disadvantaged provided they have encountered barriers to 
employment (e.g., limited English language proficiency, school 
dropouts, ex-offenders). In addition, the Act requires that recipients 
of AFOC grants and school dropouts be equitably served 1n relation to 
their Incidence 1n the eligible population. To the extent possible, 
the comparison group should only Include Individuals who meet the 
explicit eligibility criteria and who are similar to participants on 
characteristics emphasized 1n the legislation. 

10 As described 1n the evaluation guide, JTPA applicants who are 
eligible but who do not participate (commonly referred to as 
"no-shows") are also a potential source for drawing a comparison 
group. The primary advantage of using these Individuals as a 
comparison group 1s that they did go through many of the selection 
processes that participants did. Thus, they were eligible, decided to 
apply for JTPA, were selected to participate, and were assigned to a 
program activity. In addition, a major potential advantage 1s that 
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group. First, data are available on several Individual characteristics 
of Interest, including some of those related to JTPA eligibility, which 
should generally be comparably measured with JTPA HIS data and, as 
such, can be Included as control variables. 11 Second, like JTPA 
participants, new ES registrants also presumably experienced a recent 
decline 1n earnings. Finally, also like JTPA participants, ES 
registrants are 1n the labor force at the time they apply for 
assistance. As discussed 1n the evaluation guide, 1t 1s Important to 
ensure that participants and comparison group members are similar 1n 
their attachment to the labor force, or else net Impact estimates can 
be severely biased. In order to ensure comparability on pre-program 
labor force attachment, the comparison group should be drawn from new 
ES registrants 1n the same calendar quarter that participants enroll 1n 
JTPA. Following this procedure, other registrants who applied for ES 
assistance more than three months earlier and, as such, experienced 
their decline 1n earnings even earlier and who may have already 
returned to work, will be excluded from the sample frame for the 
comparison group. 

Although ES registrants have several Important advantages as a source 



because these Individuals applied to JTPA, comparable baseline data 
were collected for them, which maximizes the amount of pre-program 
Information that could be used to construct control variables for the 
net Impact model. The major disadvantages are that (1) these groups 
introduce an additional selection bias 1n that they may be 
systematically different because they chose not to participate (perhaps 
because they found a job); (2) the sample sizes of such groups may be 
too small to support precise net Impact estimates; and (3) data on 
no-shows are not usually Included 1n the JTPA MIS. Thus, although this 
approach has certain advantages, 1t raises additional questions ant 
cannot be Implemented unless states and SDAs consistently Include 
no-shows 1n the data base. 

- 11 Based on a comparison of ES and JTPA data collected 1n the State 
of Washington, 1t appears that the following Individual characteristics 
are comparably measured and could be Included 1n the net Impact model: 
age, race (white, black, Hispanic, American Indian/Alaskan Native, 
As1an/Pac1f 1c Islander) , education (whether received high school 
degree or equivalent) , handicapped status (whether has physical or 
mental Impairment that 1s a substantial handicap to employment), 
occupation (primary DOT code of previous job), veteran status (whether 
a veteran, whether a Vietnam-era veteran, whether recently separated, 
and whether a disabled veteran), Food Stamps recipient, WIN registrant, 
and economically disadvantaged status. In addition, pre-program 
measures of UI earnings, AFDC grants and whether a UI recipient will 
also be available and comparably measured for both participants and 
comparison group members. Although this 11st 1s not as complete as one 
would Ideally 1 1ke--measures of marital status, family size, dependent 
children, ex-offender status, limited English-speaking ability, and 
detailed data on pre-program employment experiences are not 
available-- 1t must be recognized that most of these characteristics 
were unavailable to previous national studies of the Impact of 
employment and training programs. As such, this 1s not a limitation 
that 1s specific to the state-level model. 



for drawrng a comparison group, a few disadvantages must also- be 
recognized. First, and most Important, because of recent reductions in 
federal reporting requirements related to the ES, states are no longer 
required to report the number of economically disadvantaged applicants 
who are registered and served by the ES. Because economically 
disadvantaged is the major criterion for JTPA el1a1b1lity, and given 
the importance of ensuring that the comparison group be similar to 
participants on all characteristics that affect eligibility, 1t Is 
important that the economically disadvantaged status variable be 
available for the net Impact model. 12 i* appears, however, that many 
states have decided to continue collecting Information on th* 
economically disadvantaged status of ES registrants, in the event that 
reporting requirements change again. States that no longer collect 
this information will have to modify the comparison group sample 
selection procedures, as described 1n Chapter 4. 

A second potential disadvantage to using ES registrants as a source for 
drawing comparison groups concerns limitations 1n procedures for 
retaining historical data, in the past, most states have kept copies 
of data tapes with Individual ES records (Including registrant 
characteristics and ES services received) for a period of three to five 
years, in some states, however, Individual- level data are purged after 
approximately one year, and backup tapes that are archived are not verj^ 
accessible. in such states, 1t will be difficult to draw the four 
quarterly samples retrospectively at one time, as comparison group 
members for the first quarter cohort would already have been purged. 
Thus, such states will have to draw the comparison samples on an 
ongoing basis (i.e., on a quarterly or semi-annual basis), alter their 
purging practices, or be sure to retain historical data for at least 18 
months to two years. 

A final potential complication 1n using ES registrants to develop a 
comparison group concerns the possibility that the ES registrant file 
may be dominated by UI claimants 1n some areas. For example, 1n states 
and local areas In which ES offices are co-located with UI offices, or 
1n which the policy is to actively monitor the Job search efforts of UI 
claimants, this could result 1n a large proportion of UI claimants in 
the ES registrant file. Because 1t is Inappropriate to compare the 
outcomes of JTPA participants with a sample that 1s dominated by UI 



It should be noted, however, that the decreased emphasis on the 
economically disadvantaged measure might also introduce additional 
measurement error Into this variable. That 1s, not only did ES staff 
previously have no r«al Incentive to accurately record the status of 
the applicant (I.e., since ES services do not depend on whether a 
person is economically disadvantaged), but they now have even less 
Incentive to do so. As a result, it 1s likely that ES offices would 
tend to underreport serving such applicants. Thus, to the extent that 
only ES registrants who are recorded as economically disadvantaged arc 
Included in the sample, their status should be measured reasonably 
accurately, which will minimize analytical complications due to 
measurement error, in the evaluation guide we discuss the biases that 
can arise because of measurement error in the dependent or Independent 
variables. 
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recipients, 1t may be necessary to undersample UI claimants 1n the ES 
registrant file 1n some states to obtain a comparison sample that has a 
proportion of claimants similar to the JTPA population. In adjusting 
for this potential problem, 1t 1s Important to note that UI claimant 
status as recorded on the HIS systems for ES and JTPA may not represent 
the same concept. In particular, for JTPA participants, being a UI 
claimant means that the person filed a UI claim and was determined to 
be monetarily eligible, whereas for the ES, claimant status refers 
simply to the filing of a claim for benefits and does not Imply 
monetary eligibility. Because of this difference, a typical JTPA 
"claimant" 1s much more likely to receive UI benefits than a typical ES 
"claimant." As a result, to ensure that UI recipiency 1s comparably 
measured for the two groups, we recommend that the UI Benefit History 
file be used to develop a measure of whether the person was a UI 
recipient as described 1n Chapter 4. A decision on the appropriate 
rate for sampling UI recipients from the ES registrant file would then 
be based on this measure. 

Despite these potential disadvantages, we believe that ES registrants 
are the best source among existing state data bases for drawing a 
comparison group to examine the net Impacts of JTPA. As a result, we 
now turn to discuss additional details related to drawing a sample of 
ES registrants, Including cases that should be excluded, and procedures 
for drawing the sample. 



Comparison Group Sample Exclusions 

Prior to selecting the comparison group of ES registrants, certain 
cases should be excluded from the sample frame to maintain 
comparability with the participant sample. In addition, 1t may be 
desirable to exclude from the comparison group Individuals who are 
clearly not eligible for JTPA and who are likely to have earnings 
potential that 1s considerably different from the earnings potential of 
JTPA participants . Below we discuss sample exclusion considerations 
concerning the comparison group of ES registrants. 

To maintain comparability with the JTPA participant sample, the group 
of ES registrants should be restricted to Individuals over 21 and under 
65 years of age. Moreover, 1f 1t turns out that no one In the JTPA 
sample 1s over, say, age 55, then the ES registrant sample should be 
similarly restricted. In addition, 1t 1s Important to exclude ES 
registrants from the sample who are JTPA participants during either the 
pre-program, inprogram, or post-program period. This problem, known as 
"comparison group contamination," results 1n comparing participants 
with other participants and yields net impact estimates that are biased 
toward zero. To minimize this problem, one should compare the social 
security numbers of current and recent JTPA participants with the 
social security numbers of ES registrants and exclude all matches from 
the comparison sample. 

A related exclusion Issue arises 1n attempting to ensure that the 
comparison group is as untreated as possible. In particular, 1n 
addition to excluding from the sample frame potential comparison group 
members who have participated in or are participating in JTPA, 1t is 
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also Important to exclude individuals who receive significant ES 
sendees." The major ES services Include referral to jobs listed 
witn the ES, job development, job counseling, and testing. Because 
Individuals who receive such- services may be receiving considerable 
employment assistance, we recommend that such Individuals be excluded 
from the comparison group sample frame. Because a substantial 
proportion of ES registrants generally receive no services other than 
registration,, there are likely to be a sufficient number of ES 
registrants who receive no services and who are similar to J1PA 
participants on measured characteristics to develop a comparison 
group. It should be noted, however, that because Individuals often 
receive some ES services a few months after they apply for assistance, 
a waiting period after application will be required before the 
comparison sample frame can be reliably purged of Individuals who 
receive potentially significant ES services. The appropriate 
procedures to be followed are described 1n Chapter 4. 

A final Important sample exclusion issue concerns procedures to ensure 
the similarity of the participants and comparison group members on 
characteristics related to eligibility for JTPA. As Indicated above 
the primary criterion for J1PA eligibility 1s that the person be 
economically disadvantaged. Although the Act requires that at least 
90% of JTPA participants be economically disadvantaged, 1n PY 84, 95% 
of adults 1n Title I I-A programs were economically disadvantaged. 
Moreover, of those who are not economically disadvantaged, or who 
cannot be certified to be economically disadvantaged, the Act requires 
that they face demonstrated employment barriers. Because virtually all 
adult Title I I-A enrollees are economically disadvantaged, and even 
those who are not certified to be economically disadvantaged may 1n 
fact be so, or may have earnings potential that 1s most similar to 
economically disadvantaged Individuals, we recommend that the 
comparison group also exclude all new ES registrants who are not 
economically disadvantaged at application. This will ensure a 
comparison group that 1s similar to JTPA participants on the key 
characteristic related to JTPA eligibility. 14 



SELECTING THE COMPARISON GROUP SAMPLE 



If ES registrants receive significant employment assistance, a 
comparison of the average post-program outcomes of J1PA participants 
and ES registrants would measure the Incremental effect of JTPA 
relative to the ES, and not the effect of JTPA per se. 

14 As described in the evaluation guide, several previous studies 
excluded from the comparison group Individuals with very high 
pre-program earnings who were clearly Ineligible to participate 1n 
employment and training programs. By matching participants and 
comparison group members on economically disadvantaged status, however, 
such additional exclusions are no longer necessary. It should be noted 
that 1f the economically disadvantaged status variable 1s not available 
1n some states for ES registrants, then procedures to exclude cases 
with high pre-program earnings would have to be Implemented as 
described 1n Chapter 4. 
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Once decisions have been made on which Individuals should be excluded 
from the comparison sample frame, the next step Involves the procedures 
to be used for drawing the comparison group. Although several 
alternative procedures for selecting the comparison group were 
considered, as Indicated 1n Exhibit 2 1t was decided to select a 
stratified random sample of ES registrants that has the same 
distribution as JTPA participants on a few key characteristics. This 
approach maintains maximum statl stlcal power to the extent possible, 
while ensuring that the participant and comparison samples are similar 
on key characteristics. 

As Indicated above, because of program eligibility considerations and 
certain practical Issues concerning the relationship between the ES and 
other programs (e.g., UI, welfare), some of the more Important 
characteristics on which to ensure comparabl 11 ty between participants 
and comparison group members are economically disadvantaged status, 
receipt of Ul, and receipt of AFDC. Because we will ensure 
comparability between the two groups on economically disadvantaged 
status by excluding from the sample frame for the comparison group all 
new ES registrants who are not disadvantaged, no additional matching 1s 
required on this characteristic. Although we do not have any 
Information on the proportion of new ES registrants that are AFDC 
recipients or that are UI recipients nationwide, there are likely to be 
relatively too few ES registrants who are AFDC recipients (e.g., 9% of 
adult men and 35% of adult women JTPA termlnees 1n PY 84 were receiving 
AFDC at application) and relatively too many ES registrants who are 
receiving UI benefits (e.g., 15% of adult men and 8% of adult women 
JTPA termlnees 1n PY 84 were UI c laV>. : nts at application). In order to 
ensure similarity on these Important characteristics, we recommend that 
comparison group members be randomly selected from the sample frames of 
adult men and women to match the distribution of participants on these 
characteristics. Thus, for the separate samples of adult men and 
women, procedures would be used to ensure that the participant and 
comparison groups are similar on the proportions In the four cells 
representing combinations of AFDC and UI recipiency status. 
Operational ly, for a given total sample size of participants and 
comparison group members, sampling rates for each cell would be 
determined to match the two distributions, and then comparison group 
members would be selected randomly from the cells at the given sampling 
rates. The definitions of AFDC and UI recipiency status to be used and 
the procedures to match the two samples are described more fully 1n 
Chapter 4. 



SAMPLE SIZES FOR THE PARTICIPANT AND COMPARISON SAMPLES 

An Important element of the research design 1s the determination of the 
appropriate sample sizes for the participant and comparison groups. As 
we Indicated earlier, 1n most states there w1 1 1 be little choice 
Involving the size of the participant sample. That 1s, all JTPA 
participants may be necessary to derive reasonably precise estimates of 
average program net Impacts. Because the marginal cost of Increasing 
sample size 1s very low (I.e., data are available 1n existing agency 
files), even 1n medium to large states one should generally use the 
largest numbers of participants and comparison group members feasible, 
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given available staff and computer resources. In states with very larqe 
JTPA programs, however, samples of participants and comparison group 
members will be drawn, which raises the Issue of total sample size as 
well as the allocation of the total sample among the two groups The 
evaluation guide contains a description of an approach that states can 
use to select appropriate sample sizes for the net Impact model. Below 
we briefly summarize the results of that discussion. 

The appropriate sample size for the net Impact model ultimately depends 
on the size of the Impact that 1s Important to detect for policy 
purposes and the level of statistical accuracy required, with larger 
sample sizes, one has greater assurance of detecting small differences 
1n overall outcomes between the participant and comparison groups, as 
well as detecting differences for major participant subgroups or across 
program activities. The likelihood of detecting a given difference 1n 
outcomes also depends on the allocation of the total sample between the 
participant and comparison groups and the unexplained variance of the 
outcome measure (I.e., earnings, AFOC grants). In general, the more 
homogeneous the samples, that 1s, the smaller the variance of the 
outcome measure, the smaller will be the number of cases necessary to 
detect a given difference 1n outcomes at a specific level of 
significance. 

Based on a number of considerations described 1n the evaluation guide, 
we Indicate that a total a nalysis sample of 12,000- divided equally 
between adult men and women, participants and comparison group members 
(I.e. 3,000 each)- -should be adequate to meet most state's analysis 
needs. It should be emphasized that this sample size recommendation 
refers to the final analysis samples and, because some cases will be 
omitted for various problems described above, Initial samples will be 
somewhat larger as described 1n Chapter 4. States that are Interested 
1n obtaining precise net Impact estimates for subgroups of adult men or 
women, or that have additional resources should consider larger sample 
sizes as necessary. Finally, states with relatively small JTPA programs 
(I.e., less than 2,000 adult enrollees per year) should be very careful 
1n Interpreting the results as only very large Impacts will be regarded 
as significantly different from zero. As a result, such states might 
consider pooling samples with other states or over time to Increase 
sample size to enhance the reliability of the net Impact findings. 

Although we believe that a total analysis sample size of 12,000 should 
generally be adequate to meet most state's analysis needs, 1t must be 
emphasized that the appropriate sample size depends on several factors 
and that there 1s no size that will be correct under all 



As described 1n the evaluation guide, with such a sample design we 
estimate that 1t will be possible to detect approximately a five (six) 
percentage point Impact on earnings for adult men (adult women) with 90% 
power at a .10 significance level. That 1s, one would have 90% power 
with a .10 significance level of detecting an overall net Increase 1n 
participants earnings of as small as five or six percentage points. The 
statistical power 1s the probability of detecting an effect at the 
chosen significance level when the effect of the specified size 1n fact 
exists (I.e., 1t 1s 1 minus the probability of making a Type II error). 
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circumstances. As such, states that are unsure as to the appropriate 
sample sizes to use 1n a net Impact analysis should review the 
evaluation guide and discuss their concerns with a sampling expert 
before making a final decision. 
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DATA COLLECTION AND PROCESSING PLAN 



In order to Implement the general research design described above, 
numerous data collection and processing tasks must be conducted. For 
example, quarterly samples of participants and comparison group members 
must be drawn, pre-program, during-program, and post-program data must 
be obtained from several sources (e.g., JTPA, ES, UI, PA) and merged to 
Individual records, the data obtained must be cleaned, certain cases 
may need to be excluded, analytic variables must be created, procedures 
must be Implemented to deal with missing data problems, and analysis 
files must be created. In this chapter we describe in detail the many 
steps Involved In a data col lection and processing cycle for the net 
Impact model and discuss potential problems that may arise. 

In the discussion below, we focus on the various data collection and 
processing tasks that must be conducted for a typical program quarter. 
In particular, we have chosen for lllust dtlon purposes the first 
quarter of PY 198b (i.e., July 1, 1985 through September 30, 1985). 
Once the quarterly data files have been created, 1t Is straightforward 
to merge these files to create a program year data file that meets the 
needs of the net Impact model. 

Although none of the data collection or processing tasks described in 
this chapter Is particularly difficult or burdensome, it Is Important 
to recognize that the overall magnitude of these tasks Is 
considerable. Moreover, many of the tasks ultimately must be performed 
by staff of several different agencies or subagencles. The size and 
breadth of the data related tasks have two Important Implications. 
First, In order to develop an effective data base system for the net 
Impact model, there must be active participation, cooperation, and 
support on the part of several state agencies and subagencles on an 
ongoing basis. Because these agencies have different priorities and 
their own policies concerning Issues such as data confidentiality, 1t 
Is Important that any Issues of concern be resolved at the outset, that 
a regular data collection and processing schedule be established, and 
that the net Impact model be given strong administrative support. 
Otherwise, the analysis may not be able to provide meaningful results 
In a timely fashion that will be of use to pol icy-makers . It must be 
emphasized that the lack of support on the part of any of the agencies 
Involved will considerably reduce the value of the net Impact results 
and could potentially render them useless. Second, In order to ensure 
that the various data processing tasks are effectively coordinated, one 
person must be given the responsibility to manage these tasks and the 
authority to obtain the necessary staff and computer assistance. 
Managing this effort is likely to be a time-consuming activity In the 
initial stages, but once the system Is In place and the individual 
tasks become routine, the management time required should dec 1 ine 
considerably. Although strong administrative support and the 
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appointment of a project manager are both necessary conditions for the 
effective management of virtually all projects, the magnitude of the 
data processing tasks and the Involvement of multiple agencies make 
these conditions particularly Important for managing the data 
processing system for the net Impact model. 

In Exhibit 3 we provide a general overview of the various data 
collection and processing tasks Involved from sample selection to 
preparing an analysis file for estimating the net Impacts of JTPA. In 
the remainder of this section, we describe the steps Involved 1n 
conducting each of these tasks. In describing the steps Involved In 
each major data processing task, we have attempted to be as specific as 
possible, and to carefully Indicate the sequential nature of the 
steps. To ensure clarity, we have tended to err on the side of 
presenting as separate steps some activities that could easily be 
combined Into one step. It should also be noted that 1n attempting to 
be very specific, we run the risk of being Incorrect. That 1s, 1t 1s 
likely that a number of problems will arise 1n conducting a state-level 
analysis that have not been fully anticipated In this Implementation 
guide. Without direct experience 1n Implementing the net Impact model 
using actual data on JTPA participants and ES registrants, and 
experience 1n merging UI Wage Records and AFDC Grants Records, 1t 1s 
not possible to fully anticipate the different kinds of problems that 
are likely to arise. Moreover, several of the recommendations provided 
below that rely on estimates of the occurrence of certain events (e.g., 
the proportion of JTPA participants or ES registrants that have certain 
characteristics) have been made without the benefit of detailed data on 
these Issues. As such, although the recommendations/guidelines 
provided below constitute our best judgments at this time, some of 
these guidelines are based on little more than "best guesses." In 
order to Improve the guidance available to subsequent users we would 
greatly appreciate being Informed of unanticipated problems that occur, 
situations that are not adequately addressed 1n the Implementation 
guide, or experiences that differ considerably from the gene al 
description provided here. 



SELECTING THE PARTICIPANT GROUPS 

As Indicated 1n Exhibit 3, the first data collection and processing 
task Involves the selection of the participant groups. As described In 
the research design above, samples of adult men and wc.nen JTPA Title 
II-A enrollees will be selected on a quarterly basis. For Illustration 
purposes, below we list steps that could be followed by JTPA data 
processing staff to select appropriate participant groups for the first 
quarter of PY 1985 (I.e., adult men and women who enrolled between July 
1, 1985 and September 30, 1985): 

1. Create a working file (on tape) of all persons who enrolled In 
JTPA Title II-A programs 1n any SDA 1n the state during the 
quarter (I.e., the first quarter of PY 1985). The file should 
Include the person's SSN, age, and sex. Although the specific 
date of enrollment Is not necessary for selecting the sample, 1t 
may be advisable to Include 1t on the working file so that one 
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Exhibit 3 

OVERVIEW OF DATA COLLECTION AND PROCESSING TASKS 



Select quarterly samples of participants and obtain, merge, and 
process pre-program data from various sources for these 
participants. 

Select expanded quarterly samples of new ES registrants who are 
economically disadvantaged to serve as potential comparison 
group members. Obtain, merge, and process pre-program data 
from various sources for the expanded samples Select 
quarterly comparison groups of adult men and women ES 
registrants who received no ES services from the expanded 
samples to match the distribution of participants on four cells 
comprising combinations of UI recipiency status and AFDC 
recipiency status. 



Create quarterly pre-program data files that Include all of the 
data obtained 1n the above steps for the samples, of 
participants and comparison group members. 



Create separate annual pre-program analysis files for adult men 
and women from the quarterly pre-program data files This 
Involves merging the quarterly files, editing the data, 
creating analytic variables, and Implementing procedures to 
handle cases with missing data. 

Obtain 1n-program and post- program outcome data (I.e., AFDC 
Grants Records and Ul Wage Records) for all quarterly samples 
of participants and comparison group members and create 
appropriate outcome variables. 

Obtain data on JTPA services for participants 1n each of the 
quarterly samples and create treatment variables. 

Create separate annual net Impact analysis files for adult men 
and women by merging the pre-program analysis files with the 
outcome and treatment variables. 
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could check to make sure that the extraction program worked 
properly (I.e., that all persons on the file enrolled 1n JTPA 
during the particular quarter of Interest and that enrollees 
during a particular part of the quarter were not Inadvertently 
omitted) . 

2. Exclude from the file all persons who are age 21 or younger, or 
age 65 or older. 

3. Stratify the participant group by sex. That 1s, create a 
subfile of adult men aged 22 to 64 and a separate subfile of 
adult women aged 22 to 64. 

4. Select a random sample of adult men and women participants from 
the two separate subfiles and determine the maximum age for men 
and women 1n the selected samples. (Information on the age 
range will be used 1n selecting comparison group members as 
described below.) The samples should be selected using the last 
two digits of the SSN, which are random numbers. The size of 
the quarterly samples should be such that the final analysis 
samples for the program year will Include at least 3,000 adult 
men and women each. In order to allow for a small--5% or so-- 
expected sample loss due to potential missing data problems 
(I.e., JTPA cases will be subsequently excluded 1f they have 
missing data on key variables such as length of stay or program 
activity), the Initial sample of adult men and women 
participants for the entire program year should be approximately 
3,200 each. To account for potential seasonal differences 1n 
JTPA enrollments (perhaps resulting from offering programs that 
only begin at certain times), one should select a fixed 
proportion of enrollees 1n each of the four quarters, and thus 
let the size of the participant group selected vary across 
quarters to reflect seasonal differences 1n enrollments. 
Because the quarterly samples must be selected long before the 
total adult enrollment 1s known for the year, however, 1t 1s not 
possible to set a fixed SSN range to generate an Initial adult 
participant sample for the program year of precisely 3,200 men 
and women each. The simplest approach, and one that should work 
reasonably well provided total enrollments are similar from year 
to year, or that enrollment changes can be reasonably well 
anticipated, would be to calculate the appropriate SSN range 
based on total adult enrollments 1n the previous year. That 1s, 
1f the state sample design calls for final analysis samples of 
3,000 adult men and 3,000 adult women over the year (or 
approximately 3,200 Initial enrollees to account for the 5% 
expected sample loss due to missing data), and 1f 8,000 adult 
men and 6,400 adult women enrolled 1n J1PA Title 1I-A programs 
1n the previous year throughout the state, then an SSN range of 
00 to 39 (I.e., a 40% sample) for men and 00 to 49 (I.e., a 50% 
sample) for women applied uniformly 1n each quarter of the 
program year should result 1n approximately the expected total 
sample size for the entire program year, although the number of 
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participants selected would vary across quarters. ^ 

Following these four steps, one could obtain a representative sample of 
adult men and women JTPA enrollees 1n the first quarter of PY 1985 By 
repeating these steps for the subsequent three quarters (and keeping 
the SSN range fixed, unless the resulting quarterly sample sizes are 
Insufficient), one could generate representative samples of 
approximately 3,200 adult men and 3,200 adult women who enrolled In 
JTPA Title II-A programs 1n PY 1985, which should be sufficient to 
generate final analysis samples of 3,000 adult men and women each. 

It should be noted that 1n using enrollments from the prior program 
year to set the SSN range 1n Step 4 above, 1t may be necessary to 
adjust estimated enrollments to reflect changes 1n real program 
resources (I.e., changes 1n federal allocations adjusted for 
Inflation). If real resources Increase (decrease), then we should 
expect enrollments to Increase (decrease), but not proportionately. In 
order to estimate the expected change 1n enrollments more precisely, 
one needs Information on the percentage change 1n program costs that 
are typically Incurred for a given percentage change 1n the number of 
JTPA participants. That 1s, one needs to know what the elasticity of 
program costs 1s with respect to the number of participants. One can 
derive an estimate of this elasticity from the cost analysis described 
1n Chapter 6. Given an estimate of the elasticity and of the 
percentage change 1n real resources, one can develop an estimate of the 
likely change 1n JTPA participants that can be used 1n setting the SSN 
range. 

Adjustments may also need to be made for changes 1n the types of 
programs offered that may affect the number of participants served. In 
making such adjustments one should carefully review SDA plans. It 1s 
also Important to monitor the overall quarterly enrollments and the 
sample sizes obtained each quarter and compare them with the quarterly 
enrollments during the previous year and the expected program year 
samples necessary for analysis. If 1t appears that the SSN range being 
used 1s not adequate to generate the necessary samples, then one should 
expand the range accordingly. 

Although these steps could be used by all states, some of the steps may 
not be necessary 1n some states and other states may need to alter the 
procedures slightly to meet their needs. For example, some states mav 
not need to stratify by sex before setting the SSN range to select the 
sample. That 1s, states that serve adult men and women 1n JTPA 1n 
approximately equal proportions could omit step (3), and set the SSN 
range to get a total Initial sample of 6,400 adults for the entire 
program year. In the Illustration used above, to get a random sample 
of 6,400 adults from a total of 14,400 enrollees, a 44% sample 1s 
required, so that an SSN range of 00 to 43 could be used. If this 



! * e " use the last two d1 9 1ts of the SSN are random numbers, 1t 
should be noted that any range or preselected set of digits will work 
equally well 1n generating representative samples. That 1s, a 50% 
sample could be generated using a range of 00 to 49, 50 to 99, 25 to 
74, or all odd or even numbers. 
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procedure were used, we would expect to end up with approximately 3,560 
men and 2,840 women (I.e., a 56/44% split), which 1s not too different 
from the recommended Initial sample of 3,200 each. Because the 
statistical precision of the net Impact analysis 1s not very sensitive 
to such relatively minor changes, given the large sample sizes 
Involved, such a state could omit the step Involving sample 
stratification by sex 1f 1t so chooses Ignoring the prestratl f 1cat1on 
by sex 1s particularly acceptable for states that serve relatively more 
men because, as we Indicated 1n the evaluation guide, for a given 
sample size for women, somewhat larger samples of men are required to 
achieve the same statistical power. Thus, as a general guide, 1f the 
population of adult JTPA participants 1n a state 1s comprised of less 
than 45% men or more than 60% men, the participant group should be 
stratified by sex before the sample 1s selected. If the proportion of 
men served 1s between this range, such prestrat1f1cat1on should be 
considered optional. However, because minimal effort 1s required to 
stratify the samples and no additional analytic complications are 
Introduced (since the models are to be estimated separately by sex), 
prestrat1f1cat1on by sex 1s recommended. 

Minor modifications to the above steps may also be made to accommodate 
states that desire larger samples, either to obtain greater statistical 
precision 1n general, or to examine net Impacts for specific subgroups 
of Interest. To obtain a larger random sample of adult men and women 
JTPA enrollees, one can simply Increase the acceptable SSN range as 
necessary within the limits of total available enrollees. For example, 
to double the sample from 3,200 to 6,400 men and women enrollees each 
per year, using the Illustration provided above one would Include all 
women enrollees and select an 80% sample of adult men (e.g., Include 
all adult men whose SSN ended 1n 00 to 79 or some equivalent range). 
If a state 1s Interested 1n oversampUng certain subgroups of adult men 
and women to Improve Its chances of detecting whether these subgroups 
benefit to a different extent from JTPA participation, 1t should 
consult a statistician/sampling expert for advice on how to select the 
sample, as well as how to correct for the differential sampling 
probabilities when examining the net Impacts of JTPA. 



OBTAINING PRE-PROGRAM DATA FOR THE PARTICIPANT GROUPS 

Once the samples of adult men and women JTPA Title II-A enrollees have 
been selected for a specific program quarter, the next step Involves 
obtaining several different data elements for these Individuals and 
merging the data onto the person's record. In particular, as we 
Indicated 1n Chapters 2 and 3, 1t 1s Important to obtain each 
participant's UI Wage Records, AFDC Grants Payment Records, and UI 
Benefit History for the pre-program period. Moreover, 1t 1s very 
Important that this Information be obtained soon after the participant 
group 1s selected. The necessity for timely acquisition of these data 
1s because (1) some states do not retain much historical UI earnings or 
AFDC grants data, and thus the sooner the data are obtained, the longer 
will be the pre-program period covered; and because (2) some of these 
pre-program data Items will be used to develop Individual 
characteristics that will 1n turn be used to select comparison group 
members that are similar on these characteristics. Below we describe 
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the various data elements that are to be obtained and merged to the 
quarterly samples of adult JTPA enrollees selected and Indicate -some of 
the problems that may be encountered 1n this process. 

T 5 e / T 1 n r . St Set nn ° f da . ta elements t0 be obtained for the quarterly samples 
of JTPA enrollees 1s their JTPA application Information. Because the 
list of SSNs used to select the quarterly participant samples was 
originally obtained from the JTPA MIS, 1t should be straightforward to 
^ n t " rn \ obta1n tne Program application Information for these 
ndlylduals For various analysis purposes, 1t will be useful to 
2u * y J , eX A raC , t - 11 appl 1cat1on data from the J1PA MIS system for 
those Individuals selected Into the quarterly participant samples. 
Although only a subset of the data Items will be used for the net 
Impact analysis (I.e., only those that can be regarded as comparably 
measured for participants and comparison group members), all 
application data should be obtained 1n case states are Interested 1n 
using the net Impact design to estimate gross Impact models, or to 
examine whether assignment to program activity depends on other 
characteristics that are available for participants, but not for 
comparison group members. After extracting the data, JTPA data 
processing staff will provide the net Impact staff with a file that 
Includes the SSN and the entire set of JTPA application data for each 
person The net Impact staff will subsequently develop measures of 
Individual characteristics from this file for Inclusion 1n the net 
Impact model . 

In addition to obtaining JlPA application data, as emphasized 1n the 
general research design discussion above, detailed pre-program earnings 
and AFDC grants histories are very Important to the net Impact model. 

, e al A 1 c y nV °" e would want at least three years of pre-program earnings 
and AFDC data, although differences In recording and data retention 
practices may make this very difficult to achieve 1n some statec To 
obtain pre-program UI Wage Records and AFDC Grants Records for the 
quarterly samples of JTPA participants, 1t 1s necessary to provide the 
appropriate agency staff with a computer tape that Includes the 3SNs of 
all adult JTPA participants (men and women) selected, along with a 
memorandum containing a clear and concise description of *he 
Information requested. Whether the tape should be ordered In ascending 
SSN or some other method depends on how the UI and PA data bases are 
organized 1n the particular state. 

There are generally six to twelve quarters of UI wages available at any 
one time, with approximately a three-month lag 1n the processing of 
these data. Thus, for example, 1f 1n October 1985, one were to request 
the quarterly UI wage records of Individuals who enrolled In JlPA 
during the first quarter of PY 1985 (I.e., between July 1, 1985 and 
September 30, 1835), one would almost always be able to obtain records 
containing quarterly wages earned since January 1984, and In some cases 
beginning as far back as July 1982. Ihe most recent wage Information 
that will have been processed by October 1985 will be for the period 
from April 1, 1985 through June 30, 1985, the Immediate pre-program 
quarter. Based on discussions with UI data processing staff 1t 
appears that although wages for prior quarters will almost always be 
complete by this time, wages for the Immediate pre-program period will' 
be updated for some Individuals after October, and will not be complete 
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until around January 1986. As a result, 1n order to ensure that the 
wages for the Immediate pre-program quarter are complete for the 
analysis, 1t will be necessary to collect data for this quarter again 
when the post-program earnings data are obtained. 

At approximately the same time, a request w1l 1 also be made for 
Information on the pre-program monthly AFDC grants received by these 
JTPA participants. In addition to obtaining pre-program monthly grants 
records for up to three years (or even longer 1f readily available), 1n 
order to define welfare recipiency status similarly for JTPA 
participants and ES registrants, we recommend that data on AFDC grants 
received for the month after the month of enrollment also be obtained 
for all Individuals at this time. Thus, as part of the Initial 
request, Af DC grants data for the month after the end of the quarter of 
enrollment of Interest (I.e., for October 1985) will also be obtained. 
Because there are generally minimal lags or delays 1n acquiring these 
data (I.e., a few weeks), this Indicates that by the middle of November 
one should be able to obtain monthly AFDC grants data from the 
pre-program period through October 1985. These monthly grants records 
will be used to create quarterly measures of welfare grants received 
that will serve as Important control variables 1n the estimation models 
described 1n Chapter 5. In addition, these monthly grants records will 
be used to create an "AFDC recipient" characteristic that 1s comparably 
defined for participants and comparison group members. To minimize 
problems caused by differences 1n the length of time from JTPA 
application to enrollment, or from differences 1n recording practices 
across SDAs, we recommend that a JTPA enrollee be defined as an AFDC 
recipient if she or he received AFOC grants during the calendar month 
prior to JTPA enrollment, the month of enrollment, or during the month 
after enrollment. A similar definition would be used to define AFDC 
recipient status for comparison group members, except that the three 
different months would be defined relative to the month of application 
to the ES. Although this definition may differ from the definition 
used 1n JTPA to define welfare recipient status, 1t will enable one to 
draw a comparison group that 1s similar on this Important 
characteristic. 

Although 1t should be quite straightforward to collect the UI Wage 
Records for the quarterly samples of JTPA participants, the PA data 
systems 1n some states may present obstacles to obtaining accurate 
pre-program AFDC Grants Records for certain types of Individuals. To 
ensure that the AFDC records accurately represent the welfare 
recipiency status of these Individuals 1n prior months to the extent 
possible, 1t will be necessary for the manager of the net Impact model 
to meet with the AFDC data processing experts to discuss situations 
that could lead to obtaining Inappropriate data for some AFDC 
recipients. For example, based on discussions with PA data processing 
experts 1n Washington State, we have Identified two situations that 
present obstacles to obtaining accurate AFDC grants paid 1n previous 
months, and these or similar problems could occur 1n other states as 
well. These two potential problems are: 

1. Although 1t 1s possible to Identify the monthly grants paid to a 
particular assistance unit for up to three prior years that a 
person with a particular SSN 1s currently 1n (using the 



"Recipient History File" that has an SSN to current assistant* 

til naH^z r ieK „ u , u possibie that KScific sisss 

was not In that particular assistance unit throughout the 
three-year period. This 1s because the history file only has 
the amount paid to the unit 1n previous months, and the SSNs of 
ndlvlduals who are currently 1n the unit and contains no 
information on who was 1n the unit 1n previous months. As a 
result the pre-program AFOC history for the unit may not 

SSI^t L^t 1 ^ < a Per !? n ^ W6lfare rec1 P 1 ency status during 
this period. This 1s particularly a problem for Individuals who 
experience a marriage or divorce, or who change I1v1na 
arrangements. " 

2. Because the data file 1s an Index-sequential file, it requires 
that the recipient can only be on the file once (1 e 1t 
requires that SSNs be unique). This can cause problems 'because 
there are two types of cases on the Washington State public 
assistance recipient file, namely AFOC and Food Stamps cases 
for example, 1f a person who receives AFOC 1s coded as a Food 

^fn P L° a K e dU , r \ n , 9 I g1ven month (P erha P s because someone moved 
into the household who had Income that was too high for AFOC but 

} 0 ?p/r n0 A U ?nV 'I, b - e 6l \ g1ble for Food Stamps >« then the entire 
3-year AFOC history 1s not available for that person. Thus 

even though the welfare system has two folders of Information on 

this particular person, the data base as designed 1s only 

capable of showing the person as either an AFDC case or a Food 

stamps case, and 1f a person on AFOC 1s shown as a Food Stamps 

case, the data base would not contain Information on the 

pre-program AFOC grants received by this person. 

of ^rHrin.n?! 3 ^ ! nformat1 °n on the pre-program AFOC status 

or participants (and comparison group members), 1t 1s necessary to 
overcome problems such as these to the extent posslb'e. To deal with 

fnuIcM r c! m . l1St6d ab0Ve ' the only Potential solution available 

P?i!< Sf?I State appears to Involve using the "Warrant Roll Extract 

1 I 0 fl cb c ™*\ n * rec ° rd °f all AFOC payments ma,3 each month and 

Lll 1n the household that month. Although these monthly 

1u eS , f are ? H Urr K ently ava1lable f0 ^ ^er 10 years (in data 
archives) 1t would be very cumbersome to use these 'lies to track 
payments to specific Individuals through tiros. It 1s nicely that the 

bl m1wh S 5JJ 1 h mPr °K em ^ ntS 1n the P re -P r °0r™ AFDC data that would 
be achieved through such a process would not he worth the hlqh costs of 
processing these numerous data files. 17 

To deal with the second problem in the State ci ngton a three 

step process 1s required, as follows: 
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oroaram rfafa Th a f V * Y< t0r , correct ^ minor errors In pre- 
program data That 1s, as discussed 1n the evaluation guide, measure- 
ment error Is a much more serious problem in the dependent variable 

^M;h 1 o P c 0S M Pr09ram measures of AF0C g^nts) than it Is In Independent 
variables (I.e., pre-program measures) . 
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1. Identify all SSNs 1n the participant group that are listed as a 
Food Stamps case (I.e., that have an "F" case record). 

2. For each SSN Identified 1n Step 1, determine whether this person 
also has an AFOC record (I.e., also has a "C" case record). 
This step Involves a manual review of the person's file folder. 

3. Obtain the monthly AFOC grants data for the subset of "C" cases 
Identified 1n Step 2, and merge them to the data file that 
contains historical AFOC grants data through October 1985. 



These three steps will enable one to overcome this limitation of the PA 
data system 1n the State of Washington and obtain accurate AFOC 
historical data for persons who enrolled 1n JTPA 1n the first quarter 
of PY 1985. After completing these additional steps, PA data 
processing staff win return a data tape to the net Impact project 
staff that Includes the SSN, the month of enrollment, and the monthly 
AFOC grants received from July 1982 (and even before 1f available) 
through October 1985. Individuals who do not match SSNs on the PA data 
base (I.e., those who received no AFOC grants during the three-year 
period) will have each of the monthly values set to zero. Similar 
procedures would be followed to obtain historical data for subsequent 
samples of quarterly enrollees and for all comparison group samples. 

In addition to obtaining historical UI Wage Records and AFOC Grants 
Records, 1t 1s also necessary to obtain historical Information on the 
UI benefits received by participants. As Indicated earlier, this 
Information will primarily be used to create a comparable definition of 
UI recipiency status for participants and comparison group members that 
will 1n turn be used 1n selecting comparison group members to ensure 
that the comparison group does not contain an excessive number of UI 
recipients. In order to develop a consistent measure of UI recipiency 
status for both groups, as discussed 1n the evaluation guide we 
recommend that a JTPA enrollee be defined as a UI recipient 1f she or 
he received UI benefits for the calendar month prior to JTPA 
enrollment, during the month of enrollment, or during the month after 
enrollment. (A similar definition would be used to define UI 
redpler.cy status for new ES registrants, except that the three 
different months would be defined 1n relation to the month of 
application to the ES. Such a definition would not be sensitive to the 
particular ES/U1 relationship 1n the state.) Although this does not 
correspond precisely to the definition used to define a UI claimant for 
the purposes of JTPA (I.e., Is eligible for UI benefits at the time of 
JTPA application, and may 1?s fact be receiving UI benefits), because 
very few participants are likely to begin receiving UI benefits after 
JTPA application, this definition should be operationally similar to 
ihe JTPA definition. Ihe following steps could be used to obtain the 
necessary UI benefits data for Individuals 1n the participant samples 
who enrolled 1n JTPA during the first quarter of PY 1985: 

1. Net Impact staff will provide UI staff with a working file that 
contains the SSN and month of JTPA enrollment for all adult men 
and women who enrolled 1n the first quarter of PY 1985 and were 
selected Into the participant samples. 



2. Ul data processing staff will compare each SSN on the working 
file with the SSNs 1n their UI Benefit History file to Identify 
Individuals who received Ul benefits for a compensable week of 
unemployment during any month from April 1985 through October 
1985. 

3. UI data processing stcff will calculate the total amount of UI 
benefits received by a person during each of the seven months of 
Interest (I.e., April through October). Monthly totals will be 
defined to Include all checks Issued for weeks 1n which the week 
ending date for which benefits were claimed 1s 1n that month, 
regardless of when the checks were Issued. All persons who do 
not match 1n Step 2 (I.e., those who received no UI benefits 
during the 7-month period) will have each of their monthly 
totals set to zero. 

4. UI data processing staff will return a data tape to the net 
Impact project staff that Includes the SSN, the month of 
enrol lment, and the monthly amount of UI benefl ts 1 ssued for 
each month from April 1985 through October 1985 for all adult 
men and women 1n the working file 1n Step 1. 

Because there 1s generally an Initial lag of a few weeks before UI 
claimants receive their first check, to ensure that all payments for 
compensable weeks 1n October have been Issued, 1t will not be possible 
to complete this task until late November or early December 1985 for 
the first quarter of PY 1985 enrollees. 

By following steps like those listed 1n each of the above pre-program 
data collection tasks, Important Information will be obtained on the 
pre-program experiences of JTPA participants from several sources. As 
the information 1s received from the various state agencies, Initial 
steps should be taken to process these data Into a format that will be 
more useful for analysis purposes. Specifically, a pre-program data 
file should be developed fcr each quarterly sample of JTPA enrollees 
that Includes the following Information: 

• Social Security number; 

• Quarter of program enrollment (e.g., 851 for those who enrolled 
1n the first quarter of PY 1985); 

• Month of program enrollment (e.g., 7 1f July, 8 if August); 

• Pre-program quarterly UI wages earned (up to twelve quarterly 
values, beginning with the Immediate pre-program quarter); 

• Pre-program quarterly AFOC grants received (calculated as the 
sum of the monthly values in a particular quarter, and beginning 
with the Immediate pre-program quarter through up to twelve 
quarters or more); 

• AFOC recipient status (defined as 1 if the person received AFDC 
grants in the month prior to the month of J1PA enrollment, 
during the month of enrol lment, or during the month after 
enrollment; 0 otherwise); 
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• UI benefits received during the Immediate pre-program quarter 
(e.g., for first quarter of PY 1985 enrollees 1t would be 
calculated as the sum of the monthly values for April through 
June 1985); 

• UI recipient status (defined as 1 1f the person received UI 
benefit payments during the month prior to the month of 
enrollment 1n JTPA, during the month of enrollment, or during 
the month after enrollment Into JTPA; 0 otherwise); and, 

• JTPA application data. 



Although the organization of the data file need not be precisely as 
that given above, 1t 1s particularly useful to place the JTPA 
application data at the end of the file. This is so that similar data 
extraction programs can be used for both participants and comparison 
group members when creating the analysis files. That 1s, because the 
same number of quarters of UI wages, AFDC grants, and UI benefits will 
be available for both groups, by ordering the data file 1n this way the 
same data Items will appear 1n the same fields on the file. The only 
difference 1n the file between the two groups would be at the end where 
the JTPA and ES application data would be placed. This pre-program 
data file will Include all of the Information necessary for selecting 
the final quarterly sample of matched comparison group members as 
described below. Moreover, all of the pre-program characteristics of 
participants that will be used in the net Impact model can be derived 
from this pre-program data file. 

Before turning to a description of how to draw the comparison groups, 
one Important feature of the pre-program data file should be noted. 
Specifically, because the file will have the same format for each group 
of quarterly enrollees, and because the pre-program data elements are 
defined in terms of their relationship to the quarter of enrol Iment 
(I.e., ordered beginning with the Immediate pre-program quarter, and up 
to 12 quarters prV r to enrollment, data permitting), data elements 1n 
the same fields will correspond to different calendar periods in 
different quarterly files. For example, although UI wages earned in 
April to June 1985 will be the first data Item in the group of wage 
Items for first quarter of PY 1985 enrollees (I.e., since 1t 
corresponds to the immediate pre-program quarter), it will be the 
fourth variable 1n the group of wage items for persons who enroll 1n 
JTPA in the fourth quarter of PY 1985. As such, before comparing 
dollar amounts in certain pre-program quarters across files, 1t is 
Important to adjust for overall price changes and translate all dollar 
amounts into real terms. Although 1t would be possible to Include the 
values of the price Index to be used for this purpose directly on the 
data file, because 1t takes on the same value for all participants and 
comparison group members 1n a particular calendar quarter, we believe 
it 1s more efficient to save space on the data file and recode the 
variables Into real terms when preparing the instructions for the 
analysis runs. This process is described in more detail later. 
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SELECTING THE COMPARISON GROUPS 



The next major data collection and processing task Involves the 
selection of the comparison groups. As Indicated 1n the research 
design above, samples of adult men and women new ES registrants will be 
selected on a quarterly basis. As such, this task will be performed by 
ES data processing staff, with guidance from the net impact project 
staff Below we 11st steps that could be used to select appropriate 
comparison groups for adult men and women who enrolled 1n JTPA during 
the first quarter of PY 1985: u 

1. Create a 11st of all ES offices that are located 1n areas served 
by SDAs 1n the state. In most cases, this will Include 
virtually all of the ES offices 1n the state. 

2. Create a working file (on tape) of all Individuals who were new 
registrants 1n the ES offices Identified 1n Step 1 during the 
period from July 1, 1985 through September 30, 1985 The 
Initial working file should Include the person's SSN age sex 
and Indicators of whether the person 1s economically 
d sadvantaged, a UI claimant, and an AFDC/welfare participant' 
Although the specific date of registration 1s not necessary for 
selecting the sample, 1t may be advisable to Include 1t on the 
Initial working file so that one could make sure that the 
extraction program worked properly. 

3. Exclude from the working file all persons who are: not 
economically disadvantaged; age 21 or less; or who are older 
than the oldest Individual 1n the quarterly participant sample. 
Thus, although the design allows for Including JTPA participants 
if they are age 64 or less, if no JTPA enrollees 1n the first 
quarter of PY 1985 are older than say age 57, the age range for 
comparison group members would be similarly restricted. 

4. Stratify the ES registrant group by sex. That 1s, create a 
subfile of adult men and a separate subfile of adult women ES 
registrants. 

5. Using the last two digits of the SSN, select an expanded Initial 
sample of adult men and women new ES registrants from the two 
S u Pa ^ te . subf1les - The size of the expanded Initial samples 
should be sufficiently large to yield quarterly samples of 
comparison groups that are at least as large as the participant 
groups 1n each quarter, after adjusting for expected sample loss 
due to excluding ES registrants who received significant 
services as well as those who are or have been JTPA 
participants. In addition, the size of the Initial samples must 
be large enough to ultimately yield sufficient numbers of ES 
registrants 1n each of the four cells defined by combinations of 
UI recipient status and AFDC recipient status, so that a 
comparison group with a similar distribution on these 
characteristics can be drawn. Because the expanded Initial 
sample size 1s likely to vary considerably from state-to-state 
depending on Individual characteristics, local economic 
conditions, and the state policies concerning the relationships 
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among the local ES, AFOC, and UI offices, U 1s very difficult 
to provide precise guidelines. As a starting point, we 
recommend that Initial samples of 2,500 adult men and 2,500 
adult women ES registrants be selected that have a similar 
distribution as the JTPA samples on the four cells comprising 
combinations of UI claimant status and AFOC/welfare 
participation status as defined on the ES data base (because 
appropriate agency data are not yet available for them). Thus, 
all persons on the two separate adult men and women subfiles 
created 1n Step 4 would be allocated to one of the four cells 
comprising combinations of UI claimant status and AFOC 
participation status and sampling rates would be determined for 
each cell to yield a random sample of 2,500 ES registrants with 
the appropriate distribution of characteristics. For example, 
suppose the sample of OTPA adult women enrollees for the first 
quarter of PY 1985 1s comprised of 8% who were UI recipients but 
not AFOC recipients, 60% who were neither UI recipients nor AFOC 
recipients, 32% who were not UI recipients but were AFOC 
recipients, and that no one was both a UI recipient and an AFOC 
recipient. Then, sampling rates should be set to yield a sample 
of 2,500 adult women ES registrants with approximately 200 
Individuals who were UI claimants but not AFOC participants, 
1,500 Individuals who were neither UI claimants nor AFOC 
participants, and 800 Individuals who were AFOC participants but 
not UI claimants, and no Individuals who were both a UI claimant 
and an AFOC participant. Thus, 1f 1n the first quarter of PY 
1985 there were 2,000 adult women ES registrants who were 
claimants but not AFDC participants, 7,500 registrants who were 
neither claimants nor AFDC participants, 1,600 registrants who 
were AFDC participants but not claimants, and 200 Individuals 
who were both a claimant and an AFOC participant, then the 
sampling rates should be set at 10%, 20%, 50%, and 0 
respectively, and the SSN ranges should be set accordingly 
(e.g., 00 to 09, 00 to 19, 00 to 49 for the first three cells 
respectively). Sampling rates for adult men would be set using 
the same approach. 

Obtain available UI Wage Records, AFOC Grants Records, and UI 
Benefit History data for all ES registrants Identified 1n Step 5 
(approximately 2,500 adult men and 2,500 adult women). It 1s 
necessary to request these data at this time 1n order to ensure 
that the same number of quarters of pre-program data are 
obtained for comparison group members as for participants. 
These data should be obtained using the identical procedures 
used to obtain pre-program data for participants as described 
above. 

Create AFDC recipient status and UI recipient status variables 
for all comparison group members on the the two separate 
subfiles using the data obtained 1n Step 6. It 1s Important 
that these variables be comparably defined to the measures 
developed for participants that were described earlier. That 
1s, comparison group members will be considered to be recipients 
of the particular Income source 1f they received Income from 
that source during the month prior to, the month of, or the 
month after ES registration. 
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8. Obtain ES services records for all Individuals Identified 1n 
Step 5 for the period from July 1 , 1985 through December 31, 
1985 (I.e., for three complete months after the end of the 
quarter of registration). Individuals will be excluded from the 
sample Identified 1n Step 5 1f they received significant ES 
services during this period. Specifically, all Individuals who 
have a transaction code on their ES service record Indicating 
that during the period from July 1, 1985 through December 31, 
1985 they received a job referral, counseling, testing, job 
development, or participated 1n a job search workshop, job 
finding club, or other similar service that may be offered 1n 
the ES offices 1n the sample, will be excluded from the samples 
of adult men and women new ES registrants. (It should be noted 
that although the length of the postregl stratlon period will 
vary somewhat across Individuals by using a fixed end date for 
receiving ES services for all registrants during a particular 
quarter, this approach was chosen to simplify data processing 
and because 1t generates a period of exposure to ES services 
that 1s approximately four and one-half months on average, which 
1s very similar to the average length of stay 1n J1PA.) 

9. Compare the SSNs of potential comparison group members who 
remain after Step 8 with the SSNs of all current J1PA 
participants, as well as with lists of participants 1n the prior 
year 1f possible (obtained from JTPA), and exclude all SSNs that 
match. That 1s, exclude from the potential comparison group 
samples all Individuals who are or who have recently been JTPA 
participants. In addition, at this point one should check to 
make sure that the remaining SSNs are unique, that 1s, that 
Individuals are 1n the comparison group only once, and 1f any 
Individuals are there multiple times, only their record with the 
earliest registration date should be retained; all others should 
be excluded. 

10. Select from the adult men and women who remain after Step 9, a 
random subset that matches the distribution of participants on 
the four cells comprising the combination of the comparable 
measures of UI recipient status and AFDC recipient status. That 
1s, the remaining SSNs on the two separate adult men and women 
subfiles would be allocated to the four cells comprising 
combinations of UI recipient status and AFDC recipient status 
(created during Step 7), and sampling rates would be determined 
for each cell to match the quarterly distribution of 
participants on these characteristics. Since all of the 
pre-program data necessary for analysis will have already been 
collected 1n Step 6, this final step should be designed to 
retain the maximum comparison sample size possible, rather than 
exclude additional cases just to reach the approximate 
participant sample size. Thus, using the above example, rather 
than setting sampling rates for each cell to generate 
approximately 800 matched ES registrants overall, and with 
approximately 64 (8%), 480 (60%), 256 (32%), and 0 registrants 
1n the four cells, the rates would be set so that approximately 
8%, 60%, 32%, and 0% of the comparison group selected are 1n the 
four cells, but retaining the maximum total sample size 
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possible. For example, 1f after the exclusions 1n Steps 8 and 
9, the 2,500 adult women Initially selected were reduced to 
1,200 1n total with 100, 800, and 300 1n the first three cells, 
the sampling rates required to obtain 800 Individuals with the 
same distribution as the quarterly participant sample (I.e., 64, 
480, and 256) would be 64%, 61%, and 85% respectively. However, 
by Including all of the Individuals 1n the cell with the highest 
required sampling rate, that 1s, by Including all 300 ES 
registrants who are AFDC recipients but not UI recipients, and 
ensuring that they remain 1n the same proportion of the total 
sample (I.e., 32%), 1t would be possible to develop a somewhat 
larger sample of approximately 938 comparison group members. To 
ensure 8% of the sample 1n the first cell and 60% 1n the 
remaining cell, this suggests total sample sizes of 75 
comparison group members who are UI recipients but not AFDC 
recipients and 563 who are neither UI recipients nor AFDC 
recipients. As such, the necessary sampling rates would be 75% 
for the first cell and approximately 70% for the second cell. 

3y following these ten steps, one could obtain samples of adult men and 
women new ES registrants 1n the first quarter of PY 1985 that are 
similar to the participant samples on Important characteristics such as 
economically disadvantaged status, AFDC recipient status, and UI 
recipient status, and who did not receive significant employment and 
training assistance. By repeating these steps for the subsequent three 
quarters one would be able to obtain sufficient numbers of matched 
comparison group members for the entire program year for analysis 
purposes. 

Although these steps could be used by most states, some of the specific 
steps may need to be modified to meet various states' analysis needs, 
data limitations, or specific circumstances. For example, states that 
face very good economic conditions could require considerably larger 
Initial samples of adult men and women than the 2,500 recommended 1n 
Step 5, 1n order to accomm<><1ate the larger-than-expected sample loss at 
Step 8 due to a greater proportion receiving ES services. Because of 
the many factors that may affect the size of the Initial samples, 
states will need to carefully monitor the sizes of the resulting 
comparison group samples to determine whether larger Initial samples 
are required. In addition, larger initial samples may be required for 
states that desire greater statistical precision. As described 1n the 
evaluation guide, one can Increase the statistical power of the overall 
net Impact estimates somewhat by Increasing the size of the comparison 
groups. Operationally, to double the expected size of the comparison 
groups, one could simply double the Initial sampling rates used 1n Step 
5 to select potential sample members from the four matched cells. 

More significant modifications to the above steps will be necessary 1n 
states that do not have all of the required data elements 1n their ES 
MIS system. The most likely potential problems relate to the 
availability of measures of economically disadvantaged status and AFDC 
participation status. As described 1n Chapter 3, because of recent 
reductions 1n federal reporting requirements, states are no longer 
required to report the number of economically disadvantaged applicants 
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who are registered and served by the ES. As a result, 1t 1s possible 
that some states no longer collect Information on whether a person 1s 
economically disadvantaged at registration. In addition, although all 
states collect some Information on whether an ES registrant 1s 
receiving public assistance at application, 1t may not be possible to 
accurately Identify whether the person 1s receiving AFDC grants at that 
point. States that face such data limitations either need to modify 
their data collection systems to Incorporate these data Items or modify 
the steps Involved 1n selecting comparison group members as described 
below. 

If the ES MIS system only Identifies public assistance recipients 1n 
general, and does not enable one to reliably Identify AFDC recipients, 
then a much larger number of Initial PA recipients must be selected at 
Step 5. The amount by which the sample of PA recipients should be 
Increased depends on the likely proportion of AFDC recipients to total 
public assistance recipients 1n the ES registrant file. It also should 
be noted that this proportion Is likely to vary between men and women. 
As an example, 1f approximately half of the women PA recipients on the 
ES file are AFDC recipients, then Instead of an Initial sample of 2,500 
adult women ES registrants of which 800 were AFDC participants, one 
would Instead draw an Initial sample of 3,300 adult women ES 
registrants with 1,600 Individuals who were PA participants 1n order to 
generate approximately 800 AFDC participants. 

A potentially much more serious problem arises 1f the ES MIS does not 
Include Information on the economically disadvantaged status of ES 
registrants. Because virtually all JTPA participants are economically 
disadvantaged, 1t 1s Important to develop procedures to select 
comparison group members who are also economically disadvantaged. In 
the absence of specific Information on economically disadvantaged 
status, an alternative approach would be to exclude Individuals with 
high pre-program earnings and then explicitly match the remaining 
comparison group members to participants on the basis of pre-program 
earnings. That 1s, Instead of excluding all persons who are not 
economically disadvantaged at Step 3, one would first obtain UI Wage 
Records for a much expanded sample at Step 6 (perhaps up to five times 
as large 1f only 20% of ES registrants are economically disadvantaged) 
and exclude all persons with high earnings 1n the Immediate pre-program 
period and who would certainly not be eligible for JTPA. Although the 
precise cut-off level 1s a matter of .judgment and depends on the 
distribution of pre-program earnings 1n both samples, a cut-off level 
set at the maximum earnings of participants (separately for adult men 
and women) 1n the six months before enrollment or somewhat higher 
(e.g., one standard deviation higher) seems reasonable. 

Although this would help to ensure similarity 1n terms of maximum 
earnings 1n the pre-program pc 1od, 1n the absence of data on 
economically disadvantaged status 1t 1s also desirable to match the 
samples more closely 1n terms of the time pattern and levels cf 
pre-program earnings. For example, based on the pre-program pattern of 
participants' earnings, one could create specific cells that are 
mutually exclusive and exhaustive, and then select comparison group 
members from these cells to match the distribution of participants. 
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For example, one could aggregate quarterly earnings Into annual 
measures for the Immediate pre-progiam years and create the following 
five cells corresponding to ranges of earnings 1n the Immediate 
pre-program year: $0, $1-$1,999, $2, 000-$4,999, $5,000-$7 r 999, and 
$8,000 to the participant maximum. To ensure that one matched the 
samples well 1n terms of recent changes 1n earnings, one could also 
create cells to identify Individuals who experienced particularly large 
decreases 1n earnings from ;he second pre-prcgram year to the Immediate 
pre-program year. For example, based on an examination of earnings 
changes 1n these years, one might define large reductions in earnings 
to be $2,000 or rcore. Then 1n addition to excluding Individuals with 
high pre-program earnings, one would Include an earnings stratification 
as part of Step 5. In this example, there wou~<d be 40 different cells 
representing combinations of earnings levels In the Immediate 
pre-p -ogram year (5 cells; , earning* changes fror.i the second to the 
first pre-program year (2 cells), UI rec^lency status (2 cells), and 
AFOC recipiency status 2 cells). By excluding Individuals with 
pre-program earnings above a particular amoun : ,nd matching the samples 
on the basis of the level and changes of pre-program earnings, one 
should be able to overcome most of the problems caused by the absence 
of Information on economically disadvantaged status. 18 It should be 
noted, however, that to ensure there are sufficient numbers of 
comparison group members to fill this many cells, considerably larger 
Initial samples may be required at Step 5. 

Two other potential data Issues should also briefly be mentioned. 
First, although the characteristics of ES registrants to be used 1n the 
steps described above are conceptually measured as of the date of 
registration, data obtained from the ES HIS reflect the current 
characteristics of the Individual at the time the data are processed. 
That 1s, 1f the person's status changes since registration (e.g., 
became a UI claimant, no longer a Food Stamps recipient), the data base 
will be updated to reflect these changes, and an Indication that the 
update was performed will be Included on the data base. Because the 



,b A potential evaluation problem remains 1n that Individuals 1n the 
comparison group with consistently zero earnings 1n the pre-program 
period are much more likely to have been 1n uncovered employment than 
otherwise similar participants who have also been Identified to be 
economically disadvantaged. Because such comparison group members are 
not economically disadvantaged, 1f they become employed In the covered 
sector, they are likely to have mwh higher earnings than participants 
1n the post-program period, which would bias downward the net Impact of 
JTPA. Because no data are available to determine for sure that 
Individuals were working 1n uncovered employment (as opposed to not 
working at all), the best one can do 1s examine to what extent the net 
Impact of JTPA varies by pre-program earnings, and, 1f the Impacts are 
large and negative for Individuals who had zero earnings 1n the 
pre-program period, this 1s likely to be the result of a selection 
bias. In this case, 1t may be better to exclude cases with zero 
earnings 1n the preprogram period from the analysis to obtain a more 
reliable estimate of the net Impacts of JTPA, It should be noted, 
however, that the resulting net Impact estimates would only be general- 
Izable to participants with positive earnings 1r, the pre-program period. 



Initial ES samples will be drawn from one to four months after the 
Individuals register with the ES, 1t 1s unlikely that the type and 
number of status changes that occur will affect the sample selection 
procedures. However, each state should review their own specific 
situation and determine whether 1t 1s Important to undertake the task 
of "undoing" the updates to retrieve characteristics measured at 
registration. 

Finally, 1t should be noted that the ability to overcome potential 
contamination of the comparison groups Is In part determined by the 
availability of data on current and prior JTPA participants. As 
Indicated 1n Step 9, 1t 1s desirable to exclude all ES registrants who 
are currently participating 1n JTPA or who participated In JTPA 1n the 
previous year. To Implement this step requires that JTPA data 
processing staff create a cumulative file of all persons (SSNs) who 
participate 1n JTPA programs and their dates of participation. If 1t 
1s not possible to Implement this step, then minor adjustments must be 
made to the overal 1 net Impact results to account for potential 
contamination bias as described In Chapter 5. 



OBTAINING PRE-PROGRAM DATA FOR THE COMPARISON GROUPS 

As Indicated above, although the steps to be used 1n obtaining various 
data for the pre- program period for comparison group members are 
Identical to the steps used for participants, the timing and magnitude 
of the task differs considerably. That Is, to ensure that the same 
numbers of quarters of pre-program data are obtained for both 
participants and the final comparison group members , as Indicated 1n 
Step 6 above, 1t 1s necessary to obtain these data for much expanded 
potential comparison group samples (I.e., approximately three times as 
large as the participant samples or larger). Moreover, because Step 5 
1n the comparison group sample selection cannot be completed until 
after the AFOC Grants and UI Benefit History data are obtained and 
processed for participants (approximately early December 1985 for first 
quarter of PY 1985 enroVlees), 1t 1s Important that Step 5 be completed 
as soon as possible after that time so that the pre-program data for 
the expanded samples selected can be obtained by the end of December 
1985 or else the comparison groups may end up with fewer quarters of 
pre-program Information. It should be noted that the same steps that 
were fol lowed to overcome potential data rel labll 1ty problems for 
participants that were discussed above must also be followed In 
obtaining pre-program data for the comparison groups. 

By the time the final comparison group members have been selected for a 
particular quarter (I.e., after Step 10), nearly all of the necessary 
pre-program Information for analysis purposes will have been obtained. 
The only exception Is the remaining ES application Information that was 
not obtained In Step 2 of the comparison group sample selection 
process. Although 1t 1s, of course, possible to obtain all of the 
application data at Step 2, because the Information 1s relevant only 
for the relatively small subset Included In the final comparison group 
samples, 1t Is somewhat Inefficient to Include these variables on the 
working files throughout. However, depending on the accessibility of 
the ES data system and the timing of demands placed on ES data 




processing staff, U may be preferable to obtain all of the ES 
characteristics at the time the Initial expanded sample 1s selected, 
but only Include the few variables needed on the working file for Step 
2, and then subsequently merge the ES characteristics data to the 
Individual records for those persons Included In the final analysis 
samples. This would avoid having to make an additional data request of 
ES each quarter. To determine the most appropriate procedures In a 
given state It will be necessary for the net Impact staff to meet with 
ES data processing staff. 

After all of the pre-program Information has been obtained, a 
pre-program data file will be developed for each quarterly sample of ES 
registrants Included In the final comparison groups. As discussed 
earlier, we recommend that the pre-program data file be organized 
similarly to the participant pre-program data file and Include: 

• Social Security number; 

• Quarter of ES registration (e.g., 851 for those who are new 
registrants 1n ES offices In the first quarter of PY 1985); 

• Month of ES registration (e.g., 7 If July, 8 If August); 

• Pre-program quarterly Ul wages earned (up to 12 quarters, 
beginning with the Immediate pre-program quarter); 

• Pre-program quarterly AFDC grants received (calculated as the 
sum of the monthly values In a particular quarter, and beginning 
with the Immediate pre-program quarter through up to twelve 
quarters or more) ; 

• AFDC recipient status; 

• Ul benefits received during the Immediate pre-program quarter 
(e.g., for first quarter of PY 1985 enrollees It would be 
calculated as the sum of the monthly values for April through 
June 1985) ; 

• Ul recipient status; and, 

• ES registration data. 

All of the pre-program characteristics of comparison group members that 
will be Included 1n the net Impact model can bt derived from this 
pre-program data file. 



PREPARING A PRE-PROGRAM ANALYSIS FILE 

Once the quarterly samples of participants and comparison group members 
have been selected and the pre-program quarterly data files have been 
developed, the next task Involves the development of pre-program 
analysis files for adult men and women. This task Involves the merging 
of quarterly data files for participants and comparison group members, 
data editing, the creation of analytic variables, and procedures for 
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handling cases with missing data. Once the pre-program analysis files 
are developed, 1t will be possible to Implement several of the analytic 
procedures described In Chapter 5 and Investigate the comparability of 
the participant and comparison groups 1n the pre-program period. Below 
we describe 1n more detail the various steps Involved 1n developing 
pre-program analysis files for adult men and women. 

Merging Quarterly Files 

The first step Involves the merging of the four quarterly participant 
data files and the four quarterly comparison group data files Into 
annual participant and annual comparison group files. Because the 
formats for each of the four data files should be Identical, the 
merging process should be straightforward. If 1t has not been done 
previously, at this point the annual files should be checked to ensure 
that the samples of comparison group members and participants are each 
unique (I.e., that no Individual 1s on one of the files more than once) 
and that there 1s no overlap between Individuals 1n the two files 
(I.e., that a comparison group member 1s not also a JTPA participant). 
Because these checks will be made by comparing SSNs either within or 
across files, this process would be more efficient 1f the files were 
first sorted by SSN. 

Editing the Data Files 

The next step Involves Implementing procedures to clean or edit the raw 
data. This process 1s basically a check on the quality and consistency 
of specific data Items. Although a considerable amount of cleaning and 
editing will have been performed by the respective ES and JTPA data 
processing staffs as part of their normal procedures, 1t will be useful 
for the net Impact data processing staff to conduct certain edit checks 
to familiarize themselves with the different files and to check the 
quality of the data. The first type of edit check Involves comparing 
the results of a simple frequency distribution on all variables 1n each 
of the annual files with a range of acceptable values. For continuous 
data elements (e.g., UI wages, UI benefits, AFOC grants), a range of 
acceptable values should be created that 1s based on common sense and 
Incorporates rough estimates of the maximum amounts that can be 
received from certain programs 1n the state. For example, one could 
recode the UI benefits received 1n the Immediate pre-program quarter 
Into several ranges such as $0; $1 -$499; $500-$999; $1 ,000-$l ,499; 
$1 ,500-$l ,999; $2,000-$2,499; $2,500-$2,999; and $3,000 or more. 
Because the highest category requires someone to have received nearly 
$250 per week 1n UI benefits, which 1s not possible 1n most states, any 
such values should be cause for additional Investigation with UI data 
processing staff to ensure that the appropriate steps were followed 1n 
creating the quarterly measure of UI benefits. 

For data Items obtained from ES or JTPA application forms, the range of 
acceptable values can be specified completely. That 1s, 1f handicapped 
status 1s coded as 1 for yes and 2 for no, then any values other than 1 
or 2 are clearly errors that likely occurred 1n entering (keypunching) 
the data Into the MIS. Thus, by reviewing a simple frequency 
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distribution of all of the data Items on each of the files and 
comparing the values to allowable codes, 1t will be possible to readily 
recognize clear data recording errors. Unless such errors can be 
readily corrected using other Information on the file, they should be 
set to a common missing data code (e.g., -9) and dealt with as part of 
the procedures for handling missing data. 

Although some errors are obvious by Inspection of a single data Item, 
other errors may not be apparent except when viewed 1n combination with 
another data Item. Thus, as a second edit check, we recommend that 
limited cross tabulations be conducted of certain Items to Identify 
additional potential data quality problems. Although such edit checks 
are usually more useful and definitive when comparing Interrelated 
Items within a given survey, 1t may be possible to Identify some 
aberrant cases 1n these agency data that should be excluded from the 
analysis . For example, cross tabulations of age by education could 
Identify 22 year-old Individuals with 19 years of education, which 1s 
quite unlikely. It would also be of Interest to cross tabulate 
earnings and AFDC grants received 1n the same pre-program quarters . 
Individuals with large values of earnings and AFDC payments 1n a given 
quarter may be Indicative of data errors, or possibly fraud. In 
addition, 1t may be Informative to cross tabulate pre-program UI wages 
with UI benefits In the Immediate pre-program period. Because 
Individuals must have worked 1n covered employment 1n the preceding 
year to year and one-half to receive UI benefits, such a cross 
tabulation could help to Identify Individuals who worked 1n uncovered 
employment (I.e., positive UI benefits but zero UI wages 1n the prior 
four to six quarters) that would be of use for the net Impact model. 



Creating Analytic Variables 

Once the data have been cleaned to the extent possible, the next step 
Involves the creation of analytic variables. In developing variables 
for the pre-program analysis file, 1t 1s very Important that the 
variables created be comparably measured for participants and 
comparison group members. For variables that are derived from a common 
source (e.g., AFDC Grants Records, UI Wage Records, UI Benefit History 
data), data comparability should not be a problem. However, measures 
of personal characteristics will be obtained from both the ES and JTPA 
HIS, and differences 1n the ways 1n which questions are asked or 
answers are recorded can present major obstacles to creating comparable 
variable definitions. In addition to the many variables that will be 
Included on the core pre-program analysis file, whl le conducting 
specific analysis tasks certain types of data may be added, general 
data transformations may be performed, and procedures for handling 
cases with missing data may be Implemented. Below we describe some of 
these steps Involved 1n developing analytic variables for a pre-program 
analysis file. 

As Indicated above, when the source of Information for participants and 
comparison group members 1s the same, the creation of analytic 
variables 1s very straightforward. In fact, the variables of this type 
that we recommend be Included 1n the pre-program analysis file will 
already be available on the pre-program annual data files at this point 




and, as such, only an extraction program 1s necessary. Specifically 
the following variables should be extracted from the pre-program annual 
data files for participants and comparison group members: 

• Pre-program quarterly UI wages earned; 

• Pre-program quarterly AFDC grants received; 

• AFDC recipient status; 

• and benef US rece1ved dur1ng the Immediate preprogram quarter; 

• UI recipient status. 

Assuming that three years of pre-program Information 1s available for 
both UI wages and AFDC grants, then 27 variables will be included on 
the analysis files for adult men and women 1n this step. 

In extracting these variables from the pre-program annual data files, 
the only modification required 1s to convert the variables that are 
expressed 1n dollar amounts Into constant dollar values (I.e., adjust 
for Inflation). Specifically, using quarterly values of the BLS 

^ S ^ e [ J! r1ce Index for 311 Urban Consumers, one would deflate 
(divide) the values of the variables expressed 1n nominal dollar terms 
by the value of the price index 1n the same calendar quarter, and 
create measures of real earnings and real AFDC grants received 1n each 
pre-program quarter and real UI benefits received In the Immediate 
pre-program quarter. It should also be noted, that although more 
aggregated measures of these variables {e.g., semiannual, annual) may 
be useful for some analysis purposes, we recommend that such 
aggregation be performed through recodlng variables while running 
specific analysis programs, rather than expanding the basic analysis 
file unnecessarily. 

Although the process Involved 1n creating consistent measures of 
analytic variables 1s straightforward when the data sources Involved 
are the same for participants and comparison group members, more 
problems can be anticipated In developing measures of personal 
characteristics that must be obtained from different agency data 
bases. The problems that can arise are most obvious when clearly 
different questions are asked, or when the answers are recorded 1n 
different ways. Moreover, even when the questions and the response 
codes appear to be the same, the Information collected may correspond 
to slightly different concepts due to differences 1n formal and 
Informal staff Instructions and training. Because of these potential 
problems, 1t 1s very Important that the net impact staff carefully 
review the application forms to both the ES and to JTPA and the 
corresponding handbooks that provide Instructions for recording answers 
to each question. Any Issues that cannot be clearly resolved through 
reviewing these materials should be discussed 1n detail with 
appropriate agency staff. 

After one has obtained a clear understanding of the personal 
characteristics in the two data bases, 1t will then be possible to 
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create comparable analytic variables. Because the application forms 
for JTPA and the ES will differ from state to state, the precise 
problems that must be confronted will differ as well. In general, we 
would expect no real problems In creating comparable measures of age, 
race, or veteran status. Provided both programs code occupation 1n the 
same way (I.e., Dictionary of Occupational Titles), there should also 
be no problem In creating comparable measures of occupation. In some 
cases, It will not be possible to create comparable variables at all, 
because one of the application forms will not obtain data related to 
the characteristics of Interest. For the State of Washington, because 
of limitations In either the ES or JTPA MIS, this currently Includes 
characteristics such as marital status, family size, presence of 
dependent children, offender status, limited English-speaking ability, 
and hours worked per week on previous job. 

Characteristics such as education and handicapped status are likely to 
fall between these two extreme situations. For example, In Washington 
State, the ES application form records the highest grade of schooling 
completed (from 0 to 19 years), whereas the JTPA application form 
records an Individual's education status In terms of one of the 
following four codes: (1) school dropout; (2) In school (high school 
or less); (3) completed high school or received GEO; and (4) currently 
attending or has attended schooling programs beyond high school . in 
terms of handicapped status, although both programs obtain some 
Information on whether a person has either a physical or mental 
disability, for JTPA the disability must constitute a substantial 
barrier to employment, whereas for the ES, the definition Is more 
general and refers to an Impairment that substantially limits one or 
more of the major life activities. Although St Is not possible to 
completely overcome the conceptual differences In handicapped status, 
It Is possible to define consistent measures of education. 
Specifically, three dummy variables can be reliably created: (1) a 
dummy for not a high school graduate (1 if response code 1 or 2 for 
JTPA enrollees and If the highest grade completed Is 0 through 11 for 
ES registrants; 0 otherwise); a high school graduate dummy (1 If code 3 
for JTPA participants and If the highest grade completed Is 12 for ES 
registrants; 0 otherwise); and a post high school dummy variable (1 If 
code 4 for JTPA participants and If the highest grade completed Is 13 
through 19 for ES registrants; 0 otherwise). Following procedures such 
as these, one should be able to obtain comparable measures of personal 
characteristics for participants and comparison group members. 

"Ihe list of personal characteristics that can be comparably defined for 
participants and comparison group members Is likely to vary somewhat 
from state to state depending on state- specif 1c data collection and 
recording practices, in general, we expect that one will be able to 
construct the following Independent variables for both groups In all 
cases: 

* Age (In years); 

* Race 
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White dummy variable 
Black dummy variable 
Hispanic dummy variable 

American Indian/Alaskan Native dummy variable 
As1an/Pac1f 1c Islander dummy variable; 

• Education 

Not a high school graduate dummy variable 

High school graduate dummy variable 

Post high school graduate dummy variable; 

• Veteran dummy variable; and, 

• Occupation (primary DOT code of previous job). 

As discussed above, depending on data availability and recording 
practices, 1t may also be possible to obtain consistent measures of 
handicapped status, marital status, family size, presence of preschool 
aged children, etc. Because of the Important role personal 
characteristics play 1n the net Impact model, every effort should be 
made to take full advantage of the data collected by JTPA and ES and 
create comparable variables whenever possible. 

The steps described above should result 1n trim pre-program analvsls 
files of 35 to 40 variables. However, 1t should be noted that 1n 
attempting to minimize the size of the pre-program analysis files 
certain variable creation activities have been set aside to be 
performed as part of the preparation activities for specific analysis 
tasks. That 1s, some variables that do not vary across Individuals 
within a site or a given quarter are more efficiently created as oart 
of the analysis runs, and other variables will be created through 
recodlng of variables already on the file. Examples of variables +hat 
do not vary across Individuals within a given site or quarter that 
should be added at this point Include: 

• Set of four quarterly dummy variables for enrolling 
(registering) 1n JTPA (ES) 1n a particular quarter of the 
program year (based on month of enrollment/registration); 

• Urban location dummy variable (based on SDA and local ES office 
locations); and, 

• Quarterly local unemployment rates (based on quarterly rates as 
they apply to the areas served by particular SDAs and ES 
offices) . 

it should be noted that although we recommend that these variables be 
created as part of specific analysis runs 1n order to minimize the size 
of the analysis files, some states may prefer to create these variables 
and attach them to the pre-program analysis records for each person. 
This will 1n part depend on the computer facilities being used for data 
processing and analysis tasks and the Importance of having trim 
analysis files that can be downloaded for analysis on a personal 
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computer. Although the first two examples above are very 
straightforward, states may find 1t particularly awkward to use a 
separate program to determine the unemployment rate for persons 1n a 
given site 1n a given quarter and then attach the data based on 
location codes. If states choose to create the unemployment rates 
separately and add them to the pre-program analysis flics for each 
person, 1t should be noted that this will add up to "12 variables to the 
pre-program analysis files. Whichever process 1s used, similar steps 
will have to be followed 1n creating post-program unemployment rates. 

Other variable creation activities can also be anticipated as part of 
the specific analysis runs. These activities will generally Involve 
recodlng variables that are already on the file to create new variables 
that enable one to Investigate a specific Issue. Such variable 
creation activities Include: 

• Create sets of dummy variables from continuous analytic 
variables. 

Create one-d1g1t DOT code dummy variables from the more 
detailed Information on the analysis files. By recodlng the 
primary DOT code Into appropriate subgroups, one would 
create separate occupation dummy variables for: 
professional , technical , and managerial workers ; clerical 
and sales workers ; service workers; agricultural , fishery, 
and forestry workers ; processing workers; machinists; 
benchworkers ; structural workers; and miscellaneous. 

Create separate age dummy variables (e.g., for ages 22 to 
24, 25 to 34, 35 to 44, 45 to 54, and 55 and older). 

Create dummy variables for whether employed 1n a particular 
quarter or whether received AFDC during a particular quarter 
based on whether the Income from the particular source was 
greater than 0 1n that quarter. 

• Aggregate quarterly variables Into semiannual or annual measures 
of earnings or AFDC grants received. Because the quarterly 
variables will already be expressed 1n real terms, this merely 
Involves summing the appropriate quarterly values. 

• Construct Interaction terms (I.e., multiply analytic variables 
together). Examples of such variables Include age squared and 
different race dummy variables multiplied by age or by different 
education dummy variables. See Chapter 5 for a discussion of 
the types of Interaction terms that will be Investigated 1n the 
net Impact models. 



Procedures for Handling Cases with Hissing Data 

In general, we do not expect that the analytic variables to be created 
will suffer from missing data problems. This 1s 1n part because of the 
data editing and cleaning that will already have been performed by ES 
and JTPA data processing staff as pa >. of their routine procedures. In 
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Instances where missing data are confronted 1n Independent variables 
however, we believe 1t 1s preferable to "patch" the variable 1n 
question rather than exclude all cases that have missing data on any 
relevant variable. Below we briefly describe the recommended approach 
for patching" cases with missing data on Independent variables to be 
Included 1n the net Impact model. 

As described 1n the evaluation guide, there are several alternative 
procedures that could be used to create substitute values of 
Independent variables for persons who for one reason or another do not 
have information on that variable (e.g., age may be missing because the 
agency staff person forgot to ask date of birth). Because we do not 
expect that missing data will be much of a problem, however, we do not 
believe 1t 1s necessary to develop en elaborate system to deal with 
such problems. As a result, we recommend that mean values of the 
Independent variables- -calculated separately for participants and 
comparison group members, and, of course, separately for adult men and 
women- be used for cases with missing data on a specific variable. 
Thus, as part of the Initial analysis tasks, one should calculate the 
means of all Independent variables separately for participants and 
comparison group members on the analysis files and prepare recode 
statements that set the value of a variable equal to the appropriate 
mean whenever 1t 1s missing. In addition, 1f there are likely to be 
differences 1n the Independent variables among the quarterly samples, 
one should consider using means calculated separately by quarter of 
enrollment/registration to capture trends 1n these variables over time. 



OBTAINING AND PROCESSING DURING PROGRAM AND POST-PROGRAM 
OUTCOME DATA 

In order to estimate the net Impact of JTPA on earnings and AFOC grants 
received, 1t 1s necessary that postenrol lment UI Wage Records and AFDC 
Grants Records be obtained for both the participant and comparison 
groups and that appropriate variables be constructed and merged onto 
the pre-program analysis file. In addition, as Indicated earlier, UI 
Wage Records must be obtained again for the Immediate pre-program 
quarter for all participants and comparison group members to correct 
for potential measurement error problems due to obtaining data "too 
early" for that period. Below we briefly outline the steps Involved 1n 
obtaining and processing these data. 

No problems are anticipated 1n collecting quarterly UI Wage Records 1n 
the postenrol lment period. In order to minimize the burden to UI data 
processing staff, provided the state maintains a reasonably long 
history of UI Wage Records, the necessary Information can be obtained 
trm a single request made at the very end of the data collection 
process. For example, assuming that length of stay 1s no longer than 
six months for all (or almost all) JTPA participants, then the first 
complete post-program quarter for the last quarter of enrol lees 1n PY 
1985 (I.e., those who enrolled between April 1, 1986 and June 30, 1986) 
will cover the period from January 1, 1987 to March 31, 1987. Because 
of the three-month time lag 1n data availability, this Information 
cannot be obtained until July 1987. In states that retain only six 
quarters of UI Wage Records, the data available 1n July 1987 will cover 
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the six ^ ">ndar quarters beginning 1n October 1985. As such, although 
this wouk< enable one to obtain all of the necessary post-program wage 
data as weM as update the earnings variable for the Immediate 
pre-program quarter for both the third and fourth quarter PY 1985 
samples of participant., and comparison group members, 1t would not 
enable one to update the Immediate pre-program UI wages for first or 
second quarter enrollees/reglstrants. In states that retain at least 
eight quarters of UI Wage Records 1t will be possible to obtain the 
required data from a single request made two years after the month 1n 
which the first quarter of Individuals were enrolled (I.e., July 1987 
for PY 1985 enrol lees); 1n states that retain only six quarters of UI 
Wage Records an Intermediate request will have to be made no later than 
18 months after the month in which the first quarter of Individuals 
were enrolled. 

After completing this process, one will have obtained eight quarters of 
UI Wage Records for participants (comparison group members) who 
enrolled (registered) during the first quarter of PY 1985, seven 
quarters of data for those who enrolled/registered during the second 
quarter, six quarters of data for those who enrolled/registered 1n the 
third quarter, and five quarters of data for those who enrolled/ 
registered during the last quarter of PY 1985. Each set of quarterly 
earnings data will Include one quarter to update the values 1n the 
Immediate pre-program quarter, one quarter for the actual quarter of 
enrollment/registration, two subsequent quarters of data that are 
likely to Include 1n-program earnings for some participants, and 
varying numbers of quarters of earnings records (from one to four) that 
can be considered to be post-program. After the quarterly records have 
been obtained, they should be divided by the consumer price Index for 
the appropriate quarter to create real earnings variables. The 
variables should then be merged to the pre-program analysis files for 
adult men and women, with the updated Immediate pre-program variable 
replacing the Initial value, and with seven additional quarterly 
earnings variables Included on each record. These seven variables will 
correspond to the real quarterly earnings during the actual quarter of 
enrollment, real earnings during each of the two subsequent quarters 
(that will usually correspond to durlng-program earnings), and four 
quarters of post-program real earnings. It should be noted that 
although the seven variables will be created for all Individuals to 
maintain the rectangular nature of the analysis file, missing data 
codes (e.g., -9's) will be placed 1n the real earnings variables for 
post-program quarters for which data are not currently available. 
Thus, for example, the second, third, and fourth post-program quarters 
will all have -9's for those who enrolled/registered during the fourth 
quarter of PY 1985. By setting these values to missing data codes 1t 
will help to ensure that one does not Inadvertently Include them 1n the 
analysis . 

With regard to collecting monthly AFOC Grants Records during the 
1n-program and post-program periods, depending on the state data system 
1t may be preferable to obtain such Information on an ongoing basis, 
rather than only once at the end of the data collection process. For 
example, for PA data systems such as those 1n Washington State 1n which 
the Recipient History File does not enable one to Identify whether a 
person 1s 1n the particular assistance unit throughout the period of 
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Interest, it would be preferable to obtain the Information on an 
ongoing basis each month to minimize measurement error In this kev 
outcome variable. As discussed above, the initial request fo? 

S^o^ 91 " 3 " 1 ^ DC grants data made 1n November 1985 for first quarter of 
PY 1985 enrollees, would cover the period from approximately July 1982 
through October 1985. For these participants and comparison group 
members, It will then be necessary to obtain monthly records from 
November 1985 through March 1987 1n order to create quarterly variables 
of AFDC grants received for the analysis. To obtain this Information 
!n^TJ g , b f S would require that for each of the next seventeen 
months, AFDC data processing staff would compare the SSNs of fVst 
quarter participants and comparison group members with the 11st of SSNs 
in assistance units that received AFDC payment? during the month The 
actual values of monthly grants would be included for SSNs that 
matched, and zeros would be included for those SSNs that did not 
match. It should be noted that the list of SSNs to be searched will 
grow every 3 months as participants and comparison group members who 
enrolled in the second, third, and fourth quarters are included. 

Although collecting postenrol lment AFDC grants data on an ongoing basis 
overcomes the major data quality issues described earlier, it imposes a 
considerable burden on PA data processing staff. If it is not possible 
to collect the data on an ongoing basis, then the Recipient History 
A ?. w1 have to be used ' and adjustments made for possible data base 
limitations discussed earlier to the extent possible (e.g., problem 
caused by assistance units that are receiving AFDC but classified as 
Food Stamps cases). Using this approach, it 1s likely that a single 
request made at the end of the data collection process would provide 
all of the necessary monthly information. 

After the monthly data are obtained, quarterly measures of real AFDC 
grants received would be created and merged to the pre-program analysis 
files for adult men and women. Once again, to maintain the rectangular 
nature of the analysis files, a common number of postenrol lment 
quarterly values will be created for all Individuals on the file, and 
-9 s will be placed 1n post-program quarters for later enrollees for 
whom the data are not yet available. 

OBTAINING AND PROCESSING JTPA MIS DATA 

The final major data collection and proces\lng task concerns obtaining 
and processing JlPA MIS data on program experiences. Because 
individuals can potentially participate in multiple activities it 
appears that most state JTPA data systems will have a program activity 
file 1n which a given Individual may have multiple records. In order 
to create consistent analytic variables to represent the program 
services received, it will be necessary for the JTPA data processing 
staff to extract the program activity records for all SSNs in the four 
quarterly samples of JTPA participants. Depending on the archiving 
procedures followed 1n a given state, it may be possible for these data 
to be obtained from a single request made at the end of the data 
collection process (I.e., around January 1987 for all PY 1985 

e ?lK } VJ' U ' however - data on PY 1985 enrollees who terminate 
within the program year are purged at the end of the program year, it 
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will be necessary to also obtain the program activity records for such 
Individuals before the records are purged (I.e., before June 30, 1986). 

After the program activity records are obtained, the next step Involves 
the creation of specific variables to represent the services received 
by J1PA participants. As discussed 1n Chapter 2, the specific 
treatment variables to be created will 1n large part be determined by 
the structure and content of the JTPA MIS. In most cases, however, 1t 
should be possible to create the following variables: 



• Participant dummy variable (1 for JTPA participants; 0 
otherwise); 

• Classroom training duiraiy variable (1 1f CT participant; 0 
otherwise); 

Remedial education and basic skills dummy variable (1 if CT 
program 1n remedial education and basic skills; 0 otherwise); 
Specific occupational skills training dummy variable (1 1f 
CT program 1n specific occupational skills; 0 otherwise); 

• 0n-the-Job-Tra1n1ng dummy variable (1 1f OJT participant; 0 
otherwise); 

• Job search assistance dummy variable (1 1f JSA participant: 0 
otherwise); 

• Other activity dummy variable (1 1f participated 1n other 
activity; 0 otherwise) ; 

• Multiple activity dummy variable (1 1f Individual participated 
1n combinations of CT, OJT, JSA, and other; 0 otherwise); 

• Occupation of training dummy variables based on one-d1g1t DOT 
code; 

Professional, technical, and managerial dummy variable (1 1f 
3-d1g1t 001 code 1s less thrn 200; 0 otherwise); 
Clerical and sales dummy variable (1 1f 3-d1g1t 00T code 
between 200 and 299; 0 otherwise); 

Service dummy variable (1 1f 3-d1g1t DOT code between 300 
and 389; 0 otherwise); 

Agricultural, f1shery v and forestry dummy variable (1 1f 
3-d1g1t 001 code 1s between 400 and 461; 0 otherwise); 
Processing dummy variable (1 1f 3-d1g1t 00T code 1s between 
500 and 599; 0 otherwise); 

Machine trades dummy variable (1 1f 3-dig1t 001 code 1s 
between 600 and 699; 0 otherwise); 

t»e*/:hwork dummy varlaole (1 1f 3-d1g1t 001 code 1s between 
700 and 799; 0 otherwise); 

Structural dummy variable (1 1f 3-dig1t 001 code is between 
300 and 899; 0 otherwise); 

Miscellaneous dummy variable (1 if 3-d1git 001 code 1s 
between 900 and 979; 0 otherwise); 
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patlon 1n weeks (calculated from 
ites); 

(1 1f participant completed JTPA 
i); and, 




• Number of hours of training per day. 

After these variables have been created, they will be merged to the 
pre-program analysis files for participants, and zeros will be entered 
for comparison group members for ail of these variables. 

As Indicated 1n Chapter 3, we recommend that participants who have 
missing data on key JTPA treatment variables be excluded from the 
analysis samples, provided the reason the Items are missing *.s random 
To make this determination, 1t will be necessary to compare ihe 
characteristics of participants who have missing data on the variables 
described above with the characteristics of participants who have data 
on the variables. For example, one should compare the ege, race, and 
education of Individuals 1n the two groups co determine 1f there are 
major differences. Moreover, 1f in the process of collecting data on 
program experiences one also obtained Information on placement status 
at termination, 1t would be useful to compare participants on their 
placement status at termination to get some sense as to whether having 
missing data 1s systematically related to the Impact of the program. 
If participants with missing data on JTPA experiences are equally 
likely to be placed 1n jobs following the program as participants with 
complete data, this would provide additional confidence that the 
validity of the analysis will not be compromised by excludlnq such 
Individuals. 



CREATING NET IMPACT ANALYSIS FILES 

Once the above data collection and processing tasks have been 
completed, the final task Involves the merging of JTPA analytic 
variables and durlng-program and post-program outcome variables to the 
pre-program analysis files. This will result 1n the creation of two 
net Impact analysis files, one for adult men, and one for adult women 
The final analysis files will Include the necessary Identifying 
Information (SSN, quarter and month of enrollment/registration) ell of 
the variables from the pre-program analysis files, the outcome 
variables, and the variables representing services provided bv JTPA 
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CHAPTER 5 
DATA ANALYSIS PLAN 



Examining the Adequacy of the Comparison Groups— Obtaining 

Evidence ®n Selection Bias 

Estimating the Average Net Impact of JTPA Programs 

Obtaining Net Impact Estimates for Various Subgroup? 

Obtaining Net Impact Estimates by Program Length of Stay 

Adjustments for Potential Data and Design Deficiencies 
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CHAPTER 5, DATA ANALYSIS PLAN 



The final step 1n developing a model to obtain valid estimates of the 
net Impacts of JTPA programs on the earnings and welfare dependency of 
adult men and women Involves the specification of a data analysis 
plan. In particular, a data analysis strategy must be developed for 
examining the adequacy of the comparison groups selected, and for using 
the comparison groups to estimate the net Impacts of JTPA on 
participants' post-program outcomes. In this section, we describe an 
overall estimation strategy for obtaining valid estimates of the net 
Impacts of JTPA. 

Before describing the details of the analysis plan, we would like to 
emphasize that the recommended approaches should be quite accessible to 
all states Interested 1n conducting net Impact analysis. For example, 
all of the analysis techniques to be used are contained In standard 
statistical software packages (e.g., SAS, SPSS) that should be readily 
available and familiar to state-level analysts. In addition, after 
some Initial data processing on a mainframe computer, 1t may be 
possible to download the analysis files to hard disks that can be 
accessed by mini-computers. This will minimize the computer resources 
required to conduct the analysis. 

An overview of the key elements of the data analysis plan 1s provided 
1n Exhibit 4. in the remainder of this section we describe the steps 
Involved 1n Implementing this plan 1n more detail. First, we describe 
an analysis strategy for examining the adequacy of the comparison 
groups selected to get a better understanding of the likely direction 
and magnitude of selection bias. We then describe a statistical model 
that can be used to estimate the average net Impacts of JTPA and the 
Impacts for Important subgroups. The section concludes with a 
discussion of potential adjustments for certain data and design 
deficiencies . 



EXAMINING THE ADEQUACY OF THE COMPARISON GROUPS- 
OBTAINING EVIDENCE ON SELECTION BIAS 

As described 1n Chapter 3, our proposed approach Involves a comparison 
of the post-program labor market experiences of a sample of J1PA 
participants with the experiences of a sample of ES registrants who do 
not receive ES services. If the samples of JTPA participants and ES 
registrants are similar on both measured (e.g., age, race, education) 
and unmeasured (e.g., attitude toward work, motivation) 
characteristics, then valid Inferences about the Impacts of J1PA 
programs can be drawn from such comparisons. However, whether an 
^dividual participates 1n JTPA or registers with the ES 1s likely to 
depend on both Individual and agency decisions. For example, J1PA 
participants must decide to apply to the program, they must meet 
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Exhibit 4 



OVERVIEW OF DATA ANALYSIS PLAN 



Examine adequacy of comparison groups using analysis 
techniques such as differences 1n means, differences 1n 
distributions, and multiple regression analysis. The 
adequacy of the comparison groups will be Judged 1n terms of 
three criteria: 

Similarity of participant and comparison groups on 
measured characteristics (e.g., age, race, education); 

Similarity of participant and comparison groups on 
pre-program earnings and AFDC grants; and, 

Similarity of the relationships between pre-program 
earnings (and AFDC grants) and measured characteristics 
for participants and comparison group members. 

Estimate average net Impacts of JTPA for adult men and women 
using an autoregressl ve model. Net Impacts will be 
estimated for four post-program outcome measures (earnings, 
whether employed, AFDC grants, and whether an AFDC 
recipient) in each of three different periods (three months, 
six months, and twelve months). 

Estimate net Impacts of J1PA on the various outcome measures 
for Important subgroups of adult men and women. Although 
autoregresslve models will primarily be used to estimate 
subgroup Impacts, 1n some cases Instrumental variable 
techniques should be considered. The subgroups of Interest 
Include: 

Participant characteristics such as race/ethn1c1ty, age, 
education, and welfare recipient status for women; 

* 

- Program activities such as CT, OJT, JSA, and other; and, 
Program length of stay. 

Adjust net Impact estimates to the extent possible for 
data/design deficiencies: 

Contamination of the comparison groups; 

Uncovered earnings; and, 

Selection bias. 
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certain legislated eligibility criteria, they must be selected by the 
agency for program participation and assigned to a program activity 
and they must decide to accept that assignment and enroll In the 
K? g fS?;«* A1th0 "9 h Es registrants do not have to meet any formal 
eligibility criteria, certain Individuals (e.g., those receiving 
benefits from government programs such as UI) are required to register 
with the ES, and some offices follow selective registration policies 
Furthermore, whether an ES registrant receives ES services depends on 
several factors, Including the availability of suitable job openings 
and the person's qualifications and persistence. Because of these 
V r ™ sele f tlon Processes, 1t 1s unlikely that the resulting samples 
of JTPA participants and ES registrants who do not receive services are 
truly equivalent on both measured and unmeasured characteristics This 
\s the Issue of selection bias. 

It 1s Important to recognize that virtually all nonexperlmental 
approaches will contain a certain amount of bias. That 1s, the formal 
conditions required to ensure unbiased estimates of program Impacts are 
not likely to be met, even 1f one had extensive data on the 
characteristics of program participants and comparison group members 
This 1s particularly true for the proposed research design, which does 
not Involve detailed survey data and, as a result, the amount of 
Information available to control for selection bias 1s limited. As a 
practical matter, therefore, one should not focus on the fact that the 
two groups are not Identical, but Identify the major dimensions on 
which tne groups differ, and determine the extent to which the net 
Impact estimates are likely to be sensitive to those differences. 

As Indicated 1n Exhibit 4, and as discussed 1n detail 1n the evaluation 
guide, we recommend that three different criteria be used to judge the 
adequacy of the comparison groups selected: 

• Similarity of the participant and comparison groups on measured 
characteristics (e.g., age, race, education); 

• Similarity of the participant and comparison groups on 
pre-program earnings and AFOC grants; and, 

• Similarity of the relationships between preprogram earnings 
(and pre-program AFOC grants) and measured characteristics for 
participants and comparison group members. 

Although these criteria are the traditional ones that have been used to 
Judge the adequacy of nonexperlmental ly-der1ved comparison groups as 
we discuss 1n the evaluation guide they are necessary, but ' not 
sufficient, conditions for the comparison groups to overcome the 
problem of selection bias. As such, even 1f the comparison groups 
selected generally meet these three criteria, this should not he 
Interpreted as definitive evidence of no selection bias, with this 
caution 1n mind, below we briefly outline some analyses that can be 
performed for each of the quarterly samples and for the annual samples 
as a whole to see whether these criteria are met, and 1f not, th;t will 
Identify the extent/types of differences between the groups that must 
be kept in mind when Interpreting the net Impact results. 
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The first criterion that we recommend be used to judge the 
comparability of the participant and comparison groups 1s the 
similarity of the two groups on measured characteristics at 
enrollment/registration. It 1s particularly Important to compare the 
participant and comparison groups on available measured characteristics 
that are known to affect earnings and AFOC grants. For example, 1t 
would be particularly useful to determine to what extent the two groups 
differ on age, race, education, occupation, and handicapped status, and 
other relevant personal characteristics that are comparably measured 
for both groups. Using standard software packages, one would compare 
the means and the distributions of these measured characteristics for 
participants and comparison group members, separately for adult men and 
women 1n each of the four quarterly samples and 1n the overall program 
year sample. 13 For example, one would compare the proportion of the 
participant and comparison groups 1n these samples that were white, 
black, Hispanic, American Indian/Alaskan Native, and As1an/Pac1f 1c 
Islander. One would also compare the average age of Individuals 1n the 
two groups and the proportions that were 1n different age ranges (e g 
22-24, 25-34, 35-44, 45-54, 550 to Identify differences 1n the "age 
distributions that may not be reflected 1n the mean age comparisons 
Because the output from these standard software analysis packages 
normally deludes the results of t-tests and Ch1-Squared tests for 
formally testing the equivalence of the means and distributions of 
variables 1n two samples, 1t 1s straightforward to compare the 
similarity of the participant and comparison groups on all measured 
characteristic*. 

We also recommend that similar analyses be conducted across JTPA 
program activities. That 1s, one should not only compare the 
characteristics of participants to the characteristics of comparison 
group members, but one should also compare the characteristics of 
participants across program activities (e.g., CT, OJT, JSa.). This 
would provide some Indication of the additional selection bias that 
could arise 1n estimating separate net Impacts by program activity. 
For example, 1f 1t was determined that more advantaged Individuals were 
being sent to OJT , the net Impacts of this program activity would be 



It would also be possible to estimate an OLS linear probability 
model of the likelihood of participating 1n JTPA to determine the major 
differences between the two groups. That 1s, one would estimate a 
regression equation with the dependent variable equal to 1 for JTPA 
participants and 0 for comparison group members, and the Independent 
variables would be all measured characteristics Included 1n the net 
Impact model described later 1n this section. This approach has the 
advantage of estimating the Independent Influence of each measured 
characteristic, while controlling for the Influence of all other 
characteristics, which eliminates the confounding effects of other 
variables that may be present when comparing mean characteristics. 
That 1s, a comparison of mean characteristics could Indicate, for 
example, that JTPA participants are more likely to be minorities and 
less educated, whereas the regression approach would account for *he 
differences 1n education by ra:e and could reveal that, after adjusting 
for differences 1n race, there are no differences between participants 
and comparison group members 1n terms of education levels. 
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somewhat Inflated because of this assignment process. That 1s the 

w^H rVe nn. P °h U1V6 m re l lat1 ° nSh ! P £ etween earn1n9S and the 0JT treatment 
would not be entirely causal, but 1n part reflect the fact that Oisl 

participants were more advantaged (I.e., more motivated). On the oth,»r 
I ' ^ I „ ther u e were re1at1 vely few differences 1n measured 
characteristics by program activity, this evidence, 1n combination with 
tes , ts ? escMbed be1ow » would provide some confidence that no 
additional selection biases would be Introduced 1n deriving estimates 
of the net Impacts by program activity. 

The second criterion we recommend be used to Judge the adequacy of the 

IrT 1S n th f s l m11aMt y of the Pre-program earnings and 

AFUC grants of participants and comparison group members This 
Involves a test of whether there 1s a significant difference 1n the 
pre-program earnings and AFDC grants of the two groups, controlling for 
m ? a fl! red chara* terlstlcs. Such a test provides valuable evidence on 
whether the two groups are comparable on the basis of the lagged 
dependent variables (after controlling for measured characteristics) 
or In other words, whether there are differences 1n the outcome 
variables between the groups 1n the pre-program period that are due to 
unmeasured characteristics. If there are any differences 1n adjusted 
pre-program earnings or AFDC grants between the two groups then this 
analysis will also provide some hints as to the direction and magnitude 
of the selection bias. For example, the extent to which J1PA 
participants have larger (smaller) adjusted pre-program earnings than 
ES registrants provides some Indication as to whether they are more 
(less) advantaged on the basis of unmeasured characteristics 
Moreover, the size of the estimated difference 1s a reasonable estimate 
? * he *™°" nt b y which the net program impacts could be overstated 
(understated) 1f the difference persisted 1n the post-program period. 

To formally test for differences 1n the pre-program earnings and AFDC 
grant-; of participants and comparison group members, one would estimate 
ord nary least squares regression equations (separately for adult m*n 
an- women) with pre-program earnings and AFDC grants as the dependent 
variables. As discussed 1n the evaluation guide, multiple regression 
s a technique that estimates the Independent Influence of each 
characteristic on a particular dependent variable, controlling for the 
Influence of all other characteristics 1n the equation. For example 
differences 1n earnings across Individuals may result from differences 
in education and other personal characteristics (e.g., age, race), 35 
well as differences In local unemployment conditions. The regret Ion 
technique controls for the Influence of local unemployment condition 
and other personal characteristics and estimates the Independent 
.nfluence of education (and all other factors) on earnings. All 
^andard .o.-i-ware analysis packages include multiple regression 
programs tnat are capable of handling the analysis tasks described 1n 
this section. 

For a given set of outcome measures, the principal task 1n specV-vinq 
the regression equations to be estimated Involves making decisions on 
which variables to Include 1n the models. Because the objective of the 
analysis 1s to Identify whether there are significant differences 1n 
the prfc-program earnings and AFDC grants of the two groups after 
controlling for measured characteristics, one should adopt the strategy 
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of Including all potentially relevant factors, except those that are 
Inappropriate for various statistical reasons. Specifically, we 
recommend that the following guidelines be used 1n making decisions 
concerning the Independent variables to be Included 1n the models: 

1. Include all personal characteristics of the Individuals at 
enrollment/registration that were examined as part of the 
analysis conducted for Criterion (1) discussed above (e.g., age, 
race, education, occupation, handicapped status), except those 
that must be omitted because too few cases exhibit that 
characteristic, or to serve as the "left-out category." For 
example, 1t 1s likely that 1n many states there will be too few 
American Indians/Alaskan Natives and As1ans/Pac1f 1c Islanders 
(and 1n some states too few H1span1cs) to Include these as 
separate variables 1n the model. As a result, one may need to 
collapse the five radal/ethnlc group variables Into three 
variables: white dummy, black dummy, and other racial group 
dummy. It 1s also Important to note that 1n estimating the 
regression model one of the race dummies must be omitted to 
serve as the "left-out category" or reference category for 
comparison purposes. For example, 1f the white dummy variable 
1s omitted from the equation, then the coefficients of the other 
two race variables would represent the effect of being 1n that 
group relative to being white. It should be noted that for 
every set of dummy variables that are Included 1n the regression 
model to capture the effects of a certain characteristic, one of 
the variables must be omitted to serve as the reference category 
for comparison purposes. Because the effects of the Included 
variables are all measured r elative to the left-out category, 
the results have the Identical Interpretation no matter which 
variable 1s chosen to serve as the left-out category. 

2. Include all prio r quarterly earnings and AFOC grants variables. 
That 1s, 1n examining the comparability of earnings 1n the 
Immediate pre-program quarter, one should Include quarterly 
earnings and AFDC grants from the second through the twelfth 
pre-program quarters, given data availability. If, however, one 
were examining the comparability 0 f. earnings and AFOC grants 1n 
the Immediate pre-p; jgrafc! year, then the second, third, and 
fourth pre-program quarterly earnings and AFOC grants variables 
would have to be omitted from the regression equation. This 1s, 
of course, because such variables are definitional ly part of the 
dependent variable 1.* this case and, as such, cannot 
Independently affect Its value. 

3. Exclude variables that are "jointly determined" with pre-program 
earnings and AFOC. Specifically, the following three variables 
described 1n Chapter 4 should be excluded from all regression 
equations estimated over a pre-program period: AFOC recipient 
dummy, UI recipient dummy, and UI benefits 1n the Immediate 
pre-program quarter. 

4. One should also consider Including potential "Interaction" 
effects among various variables 1n the model . Interaction 
variables are simply the product of two (or more) variables 
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that are already entered separately 1n the model. They are 
included to allow the effect of a particular variable to vary 
depending on the value of another variable. For example, the 
effect of education on earnings may vary depending upon whether 
the person 1s white, black, or other; additional variables 
defined as the product of race dummies and education dummies 
would allow one to test for such a relationship. As discussed 
1n the evaluation guide, because previous employment and 
training program evaluations often estimated entirely separate 
models by race, 1t will be particularly Important to Investigate 
whether the effect of other variables 1n the model (e.g., age, 
education, prior earnings) differs by race. 

5. Finally, 1n selecting the variables to be Included 1n the 
regression models, 1t 1s Important that they be defined 
according to the appropriate time period. For example, 1f the 
dependent variable 1s earnings or AFDC grants 1n the Immediate 
pre-program quarter (year) then the unemployment rate 1n the 
local area should similarly be defined as the rate for the 
Immediate pre-program quarter (year). 



By following the guidelines provided above, one could Identify a set of 
Independent variables from those created using the procedures described 
1n Chapter 4 that should be Included 1n both the pre-progra.n earnings 
and AFOC grants equations. Based on these guidelines, below we provide 
an example of a set of Independent variables that should be Included 1n 
regression equations for adult men and women to examine the similarity 
of participants and comparison group members on earnings and AFDC 
grants in the Immediate pre-program quarter: 

• Age ; 

• Age squared; 

• Black dummy; 

• Other non-white dummy; 

• High school graduate dummy; 

• Post high school education dummy; 

• (Age) x (white dummy); 

• (Age squared) x (white dummy); 

• (High school graduate dummy) x (white dummy); 

• (Post high school education dummy) x (white dummy); 

• Veteran dummy (for men only); 
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• Set of eight one-d1g1t DOT dummies (e.g., allowing professionals 
to be the left-out category, the eight occupation dummies would 
correspond to clerical and sales; service; agricultural , 
fishery, and forestry; processing; machine trades; benchwork; 
structural work; and miscellaneous); 

• Handicapped dummy (1f measured comparably for both groups); 

• Prior quarterly earnings (I.e., pre-program quarters 2 through 
12, data permitting) ; 

• Prior quarterly AFDC grants (I.e., pre-program quarters 2 
through 12, data permitting); 

• Unemployment rate during the Immediate pre-program quarter; 

• Urban location dummy; 

• Set of dummies for the quarter of enrollment/registration (e.g., 
allowing the first quarter to serve as the left-out category, 
dummy variables for whether participants (comparison group 
members) enrolled (registered) 1n quarter 2, 3, or 4); and, 

• JTPA participant dummy. 



Given the above set of Independent variables, the test for differences 
1n earnings and AFDC grants between the participant and comparison 
groups 1n the Immediate pre-program quarter would be based on a t-test 
of the estimated coefficient of the JTPA participant dummy variable. 
The t-rat1o 1s given by the estimated regression coefficient divided by 
Its standard error, and Is provided as part of the printed output from 
all major software statistical packages. Given the large sample sizes 
Involved, the hypothesis of no difference between the two groups 1n 
pre-program earnings or AFDC grants would be rejected at the .05 (.01) 
significance level 1f the calculated t-rat1o exceeded 1.26 (2.58) 1n 
absolute value. 20 On a more Intuitive level, because participation 
in JTPA during a given period can not have an effect on earnings or 
AFDC grants 1n previous time periods, the coefficient of the JTPA dummy 
variable 1n each of the regression models described above should be 
zero. The extent to which the estimated coefficients deviate from zero 
provides evidence on the direction and magnitude of the likely 
selection bias. That 1s, with earnings 1n the pre-program period as 



dU One could also estimate a similar pre-program earnings or AFDC 
grants equation and replace the JTPA participation dummy variable with 
a set of variables representing different program activities to provide 
Information on the differences 1n the pre-program outcome measures 
across program activities that are due to unmeasured characteristics. 
By testing whether these separate program activity coefficients are 
significantly different from zero, this would Indicate whether there 
are likely to be any additional selection biases 1n the assignment of 
programs to Individuals that will affect the net impact estimates by 
program activity that are described later 1n this section. 
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the dependent variable, large negative (positive) coefficients on the 
JTPA dummy would Indicate that participants were less (more) advantaged 
than comparison group members 1n the pre-program period on unmeasured 
characteristics, and 1f this persisted through the post-program period 
it would likely result 1n understating (overstating) the net Impact of 
JTPA on earnings. Thus, for example, 1f this analysis Indicated that 
after adjusting for differences 1n measured characteristics the 
pre-program earnings of JTPA participants were $200 less (more)' than 
the earnings of the comparison group, then one might consider addlnq 
(subtracting) $200 to (from) the net Impact estimate to adjust for 
differences 1n unmeasured characteristics. It should be noted 
however, that because pre-program earnings and AFDC grants will be 
Included as Independent variables 1n the net Impact model, the extent 
of this bias should be less 1n the post-program period. As such 
adjusting the net . Impact estimate for the total difference 1n 
pre-program earnings 1s likely to overcompensate for the bias due to 
differences 1n unmeasured characteristics. 

In analyzing the pre-program similarity of earnings and AFDC grants 
between the two groups, 1t 1s possible to examine several different 
time periods. For the most part, one should be primarily Interested 1n 
examining the Immediate pre-program quarter or year, and separate 
regression equations like the one descrloed above could be estimated 
for both periods.-" In addition, one could also estimate a 
regression equation like the one described above for each pre-program 
quarter and derive a set of estimated coefficients of the J1PA 
participant dummy. To the extent that Including additional lagged 
values of earnings and AFDC grants 1n the equation serves to reduce the 
differences between the two groups, the coefficients of the JTPA dummy 
variable should be largest (1n absolute value) 1n the early pre-program 
periods and tend toward zero as the pre-program outcome 1s measured 
closer to the date of enrollment. 

The third criterion that we recommend be used to judge the adequacy of 
the comparison groups 1s the similarity of the relationships between 
earnings (and AFDC grants) and Individual characteristics for JTPA 
participants and for comparison group members 1n the pre-program 
period. This criterion, which 1s considerably more strict than the 
previous two, 1s quite Important because, 1f the same model 1s 
generating earnings (or AFDC grants) 1n the two groups, 1t suggests 
that program Impacts will be less sensitive to other potential 
statistical problems. This would provide additional confidence 1n our 
ability to obtain unbiased estimates of program Impacts. 

To test for differences 1n the pre-program earnings (or AFDC grants) 
equations of participants and comparison group members, one would 

Va As ^ 1nd1cated 1n the second and fifth guidelines for selecting 
independent variables discussed above, the only changes necessary in 
the independent variables 1n changing the dependent variable from the 
Immediate pre-program quarter to the Immediate C"o-program year would 
?k< 1° ens " re that quarterly earnings and AFDC g-ents 1n the second, 
third, and fourth pre-program quarters were excluded and that the 
unemployment rate was defined for the entire pre- program year rather 
than just for the immediate pre-program quarter. 
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estimate a modified version of the regression equation that was 
described above to provide Information on Criterion (2). Specifically 
one would estimate an equation that Included all of the explanatory 
variables listed above, plus each of the variables (except the JTPA 
participation dummy variable) multiplied by the JTPA participation 
dummy variable. The formal test of whether the earnings and AFDC 
grants equations 1n the pre-program period are different for 
participants and comparison group members 1s based on an F-test of the 
hypothesis that the coefficients of the Interaction terms (1.e , the 
JTPA participant dummy multiplied by each of the other variable's Vn the 
model) are all zero. This 1s 1s sometimes referred to as a "Chow" 
test. Host standard regression programs will allow one to perform an 
F-test of such an hypothesis and will also provide the calculated 
F-stat1st1c for the test. 22 

Under the assumption that the error terms are normally distributed the 
test statistic follows Snedecor's F-d1str1but1on with r degrees of 
freedom 1n the numerator and N-K degrees of freedom 1n the 
denominator, where r 1s the number of restrictions being tested (1 e 
the number of Independent variables that have been multiplied by the 
JTPA dummy), and N-K 1s the number of degrees of freedom when no 
restrictions are Imposed (I.e., total sample size less the number of 
variables 1n the equation). If the test statistic exceeded the 
critical value for the specified level of significance, then the null 
hypothesis would be rejected (I.e., we would conclude that the 
pre-program earnings (or AFDC grants) equations for the two groups are 
not similar). For example, suppose the pre-program earnings equation 
for adult men had 30 Independent variables and a total sample size of 
6,000. Then, there would be 30 degrees of freedom 1n the numerator and 
5,940 degrees of freedom 1n the denominator, so the test statistic 
would follow an F(30, 5,940) distribution. Given the critical values 
for the F-d1str1but1on with these degrees of freedom, one would 
conclude that the earnings models of the two grouos were different at a 
.05 (.01) significance level 1f the test statistic exceeded 1.46 (1.69). 

The three criteria and related analyses described above should provide 
considerable Information regarding the adequacy of the comparison 
groups 1n the pre-program period and the types of likely biases that 
must be dealt with. It should be emphasized again, that these criteria 
are relatively strict tests of the comparability of the two groups and 
one should not generally expect nonexperlmental ly-der1 ved comparison 
groups to meet all of them. If ^he conditions *re generally satisfied, 
then the chances of obtaining unbiased program net Impact estimates 
using the standard statistical models described below are considerably 
Improved. If the criteria are strongly rejected (e.g., F-stat1st1cs of 
10 or 20 when approximately 1.5 1s sufficient for rejection), then one 
should be very careful 1n proceeding to estimate net Impacts with these 
comparison groups. Instead, one should first double check to ensure 
that the data processing and analysis guidelines described earlier were 
followed and, 1f the criteria are still strongly rejected, one should 
then consider obtaining assistance from a researcher familiar with 



If the regression packages available to a state do not have this 
feature, one should consult the evaluation guide where an alternative 
approach to performing this test 1s described. 
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these Issues. If, as Is most likely, the results are somewhere 1n 
between (i.e., pre-program differences between the two groups that are 
sometimes statistically significant, but not exceptionally larg») then 

?!J.rnr L *° " nde !\ stand the ^plications of these differences for 
Interpreting and adjusting the net Impacts results. 

ESTIMATING THE AVERAGE NET IMPACTS OF JTPA PROGRAMS 

We now turn to a description of the approach to be used to estimate the 
average net Impacts of JTPA programs on participants' post-program 
outcomes. As described earlier, the four general post-program outcome 
measures are earnings, whether employed, AFOC grants, and whether an 

k? k*I P U,U - . Bel0W We d1scuss the specific post-program periods for 
which these outcomes will be measured for different samples of 
enrollees and describe the regression model to be used to estimate 
average net Impacts. A subsequent section describes how to obtain 
separate estimates of net Impacts for major demographic groups, bv 
program activity, and by length of program participation. 

Choice of Post-program Periods and Implications for 
Potential Additional Sample Exclusions 

As discussed 1n the evaluation guide, the choice of the post-program 
2? r !°?! A 0 be exam1ned for samples of JTPA enrollees depends on the 
distribution of length of stay in JTPA. For example, if no Individuals 
partlc.pated in JTPA longer than six months, then for a given quarterly 
ofVnoc? enrollee u s (e.g., those who enrolled during first quarter of 
PY 1985), all such individuals would have terminated from the program 
by the end of the third quarter of PY 1985. As such, earnings and AFDC 
grants received during the fourth quarter of PY 1985 would be the 
outcome measures for the first complete post-program quarter for these 
enrollees If, however, there is considerable diversity 1n program 
length of stay and some individuals remain 1n the program much longer 
one would have to decide whether to postpone the analysis and wait 
untl all cases had terminated, or exclude such cases from the analysis 
samples. Although it 1s generally not desirable tc restrict the 
participant sample to those who have terminated from Jtpa by a 
particular date (because terminees could differ systematically from 
nonterminees which could result in additional selection biases) in 
most cases U will simply not be possible to wait for all participants 
results" fr ° m program and st111 Provide timely net impact 

In order to provide timely results, we recommend that individual states 
choose a cut-off date that defines the 1n-program period, and that any 
participants who are in the program after that point be excluded from 
the analysis. " In general, we expect that defining the cut-off date 

22 It should be noted that individuals who are still in the program 
in a given quarter should not be included when analyzing the impact of 
JlPA on earnings or AFOC grants during that quarter because their 
earnings may be unusually low (e.g., for classroom training or job 
search assistance participants) or unusually high (e.g., for OJT 
participants), which would bias the estimated overall net Impacts 
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of program participation to be six months after the end of the period 
of enrollment for each quarterly sample (e.g., March 31, 1986, for 
those who enrolled 1n JTPA during the first quarter of ?Y 1985) should 
be adequate to meet most states' needs. This allows for a proqram 
length of stay that 1s no less than six months for any Individual' and 
up t o nine months for Individuals who enrolled very early 1n a 
particular quarter. We exnect that such cut-off dates, applied 
uniformly to participants 1n all four quarters of the program year 
would result 1n excluding no more than ten percent of the participant 
sample 1n most states, which 1s unlikely to significantly bias the 
average net Impacts of JTPA, and should also not significantly reduce 
the precision of the estimated Impacts. For states that operate ^JTPA 
programs that tend to have very long program lengths of stay they 
should consider extending the cut-off date, which will require an 
additional data collection effort and delayed results 1n order to 
estimate earnings Impacts for the same number of post-proqram 
quarters. On the other hand, for states 1n which the JTPA services are 
relatively brief on average, It may be possible to define a cut-off 
date that allows for a shorter 1n-program period and, as a result, net 
Impacts can be estimated over a longer program period without delaylnq 
the analysis. In the Ideal situation, a cut-off date of six months 
after the end of the quarter of enrollment will result in excluding no 
(or only a minimal number of) participants from the analysis samples. 

As described 1n Chapter 2, we recommend that the net Impacts of JTPA 
for adult men and women be estimated on each of the four general 
outcome measures for three different post- program calendar periods 
Specifically, post-program net Impacts are to be estimated for a 
,hree-month period, for a six-month period, and for a twelve-month 
period. Based on the data collection plan described In Chapter 4 and 
the strategy to be used to exclude long-term stayers (1f necessary) 
the research design will enable one to estimate the net Impacts of JTPA 
on these four measures over a three-month post-program period for JTPA 
enrcllees from all four quarterly samples; 1t will enable one to 
provide net Impact estimates on these outcomes measured over a 
six-month post-program period for the first three quarterly enrollment 
samples; and 1t will enable one to estimate the net Impacts for a 
twelve-month post-program period only for participants who enrolled 1n 
the first quarter of the program year. Because o; the different sample 
sizes Involved 1n a--lyz1ng Impacts for different post-program periods 
the precision of t- estimated annual net Impacts will be much less 
than the precision r: the estimated Impacts over a three-month period 
Because of the Importance of longer-term Impacts 1n making Judgments 
concerning the effectiveness of employment and training programs, some 
states might consider expanding the research design to collect 
additional quarters of post-program Information for Individuals who 
enrolled 1n the last three quarters of the program yea>\ It must be 
recognized, however, that the final analysis results would necessarily 
be delayed almost another year. 



Autoregressive Net Impact. Models 

As described 1n the evaluation guide, to estimate the net Impacts of 
JlPA for adult men and women, we -er^end that autoregressive models 
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be used. Using this approach, ordinary least squares rf>oress1on 
equations would be estimated for each of the twelve outcome variables 
that Is, four outcome measures 1n each of three different p^ fc-orogram 
periods. The underlying philosophy of the model 1s to control as much 
as possible for potential differences in the characteristics of 
participants and comparison group members. As such, tha Independent 
variables should Include all pre-program characteristics that are 
comparably measured for participants and comparison group members In 
particular, the approach derives Its name (I.e., "autoregressi ve") 
because pre-program values of the outcome measures (i.e, quarterly 
earnings and AFOC grants) are also Included as Independent variables 
Although this results 1n a rather Inelegant net Impact estimation 
equation, 1t has the primary advantage of controlling for any 
differences 1n measured characteristics between the two groups that 
remain after the matched comparison groups are selected, which helps to 
minimize the problem of selection bias. 

To control for potential differences 1n the characteristics of 
parties, -nd comparison group members to the extent possible, we 
recomr. • • vt the net Impact regression model be a slightly expanded 
versloi .v rue models used to determine whether the comparison groups 
meet tht |,re-program comparability Criteria (2) and (3) above. The 
only changes 1n the Independent variables to be Included 1n the basic 
nec Impact model as compared to the versions of the models discussed 
above are as f ol lows : 

1. Quarterly earnings and AFDC grants 1n the Immediate pre-program 
quarter should be Included 1n the net Impact model. 

2. The net Impact model should also Include the UI recipient dummy 
variable, UI benefits earned 1n the Immediate pre-program 
quarter, and the AFDC recipient dummy variable. 

3. The unemployment rate should be defined according to the 
post-program period for which the model 1s being estimated. 

Thus following these guidelines, an example of an Initial net Impact 
mooel for earnings and AFDC grants for a three-month post-program 
period would Include the following Independent variables: 

Age; 

Age squared; 
Black dumrr.y; 
Other non-white dummy; 
High school graduate dummy; 
Post high school education dummy; 
(Age) x (white dummy); 
(Age squared) x (white dummy); 
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• (High school graduate dummy) x (white dummy); 

• (Post high school education dummy) x (white dummy); 

• Veteran dummy (for men only); 

• Set of eight one-d1g1t DOT dummies (e.g., allowing professionals 
to be the left-out category, the eight occupation dummies would 
correspond to clerical and sales; service; agricultural, 
f1s;iw>ry, and forestry; processing; machine trades; benchwork' 
structural work; and miscellaneous); 

• Handicapped dummy (1f measured comparably for both groups); 

• Pre-program quarterly earnings (I.e., pre-program quarters 1 
through 12, data permitting); 

• Pre-prograr.i quarterly AFOC grants (I.e., pre-program quarters 1 
through 12, data permitting); 

• AFOC recipient dummy; 

• UI redp.ent dummy; 

• UI benefits , jcelved 1n the Immediate pre-program quarter; 

• Unemployment rate during the post-program three-month period; 

• Urban location dummy; 

• Set of dummies for the quarter of enrollment/registration (e.g., 
allowing the first quarter to serve as the left-out category] 
dummy variables for whether participants "(comparison group 
members) enrolled (registered) 1n quarters 2, 3, or 4); and, 

• JTPA participant dummy. 



With such an autoregressl ve model estimated separately for adult men 
and women, the estimated coefficient of the J1PA participant dummy 
variable represents the average net Impact of JTPA on earnings and AFOC 
grants for a three-month post-program period for both groups. Similar 
autoregresslve models would be estimated for the six-month and 
twelve-month post-program periods for adult men and women, and the 
estimated coefficient of the J1PA participant dummy variable 1n these 
models will also provide an estimate of the average net Impact of JTPA 
on the post-program outcomes during these periods. For dependent 
variables that are expressed 1n dollar terms (i.e., earnings and AFOC 
grants), the coefficient of the J1PA participant dummy variable can be 
Interpreted as the average d ollar Impact on a given outcome measure 
By d1v1d1nn the estimated dollar Impact by the mean earnings or AFOC 
grants of comparison group members, this provides an estimate of the 
percentage change 1n earnings or AFOC grants due to JTPA. 

For dummy, dependent variables (I.e., whether employed 1n a particular 



79 

84 



ERIC 



period, or whether receiving AFOC grants during a particular oerlodl 

J h6 K K^ 0 : e9reS , S \ Ve net 1mpact mod&1 1s equivalent to a* i f 
Probability model. That 1s, the mociel essentially estimate s the 
effects of various factors on the probability of a certain event 
occurring (e.g. having positive earnings 1n a given post-program 
period). As such, the estimated coefficient of the JTPA participant 
dummy variable can be Interpreted as the average percentage pnin t 
change 1n the probability of working or receiving AF DC grants due to 
JTPA. Once again, by dividing the estimated percentage point change by 
the mean proportion of comparison group members, one can obtain an 
estimate of the percentage change 1n the probability of working (or 
receiving AFDC) due to JTPA. a K 

OBTAINING NET IMPACT ESTIMATES FOR VARIOUS SUBGROUPS 

The models' described above have focused on providing overall estimates 
of the net Impacts of JTPA for adult snsn and women. It 1s also of 
considerable policy Importance to deterge whether the effectiveness 
of JTPA varies by the type of program activity or by personal 
characteristics Because these factors (I.e., JTPA program activities 
and participant characteristics) may change considerably over time 
knowledge of how program net Impacts vary among them would help 
Interpret time trends 1n JTPA's Impacts. Furthermore, Information on 
wh ch programs work best for which types of participants could provide 
valuable Information for targeting future employment and training 
programs to the disadvantaged. Although the approach to estimating net 
impacts for different subgroups 1s formally Identical, whether the 
subgroup refers to the type of program activity or to Individual 
characteristics, additional selection biases are likely to arise 1n 
some situations. Below we describe how to modify the autoregressl ve 
earnings and AFDC grants models described above to estimate the net 
impacts of JTPA for various subgroups and Indicate the additional 
Diases that one must be aware of 1n Interpreting the results. 

In general, subgroup effects are estimated by Including 1n the 
regression equation an Interaction term that represents the product of 
the dummy variable for JTPA participation with the variable for the 
subgroup of Interest. Suppose one 1s Interested 1n testing whether the 
net Impact varies by a characteristic that 1s represented by the three 
dummy variables Z-, , Z 2 , and Z 3 . For example, one might think of 
the three variables as representing race/ethnldty categories (white 
°A her) ' or P roaram services (CT, 0J1, JSAJ.24 Then tne only 
modi 1cat1on required to the autoregressl ve model described I above 
Involves replacing the JTPA participation dummy variable with three 

!!r f n,L S H h °r ld , b ! n ° s ted Ml at ' 1n Principle, similar analyses could be 
performed to determine wither net Impacts vary across local labor 
market conditions. Howevir, because the labor market variables would 
take on the same value f;.r all persons 1n the same local area 1n a 
given time period, there :s not likely to be sufficient variation 1n 
these variables to obtain yeclse estimates of how program Impacts vary 
across local labor market conditions, except 1n large states, with many 
SDAs, and where there are considerable differences 1n labor market 
conditions across SDAs. 
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variables that each Involve the J7PA dummy variable multiplied by one 
of the three variables representing the particular subgroup (I.e., JTPA 
x Z lf JTPA x Z 2 , JTPA x Z3). The coefficients of these three 
variables are estimates of the net Impact for the three subgroups of 
Interest. To formally test whether the program net Impacts differ 
significantly across the subgroups of Interest, an F-test 1s used. In 
this case, the test statistic follows an F(r,N-K) distribution, where r 
1s the number of restrictions Imposed by the basic model (equal to the 
number of subgroups minus one), and N-K 1s the number of degrees of 
freedom 1n the basic Impact model. The null hypothesis that the net 
Impacts do not vary across the subgroups of Interest (e.g., across 
racial groups) would be rejected for r=2 and sufficiently large sample 
sizes at the .05 ( .01) significance level 1f the test statistic 
exceeded 2.99 (4.60). Most standard software analysis packages can 
calculate this F-stat1st1c as part of the analysis run. 

It should be noted that 1n attempting to disaggregate JTPA net Impacts 
across subgroups, 1t 1s Important that the subgroup characteristics 
also be Included 1n the model as control variables to account for 
differences 1n the general level of earnings (or AFDC grants) across 
these subgroups. That 1s, 1n our Illustration, the three 2^ 
variables must also be 1n the model separately so that the estimated 
net Impact coefficients only capture outcome differences due to JTPA 
across these subgroups and do not also capture the average differences 
1n outcomes due to the Z^'s themselves. In addition, 1t 1s also 
Important that the subgroups be mutually exclusive and exhaustive For 
example, suppose the Z^'s refer to various participant age 
categories: 22-34, 35-54, and age 55-64. Then two types o\ problems 
can arise 1n estimating the net Impacts for these aje groups: (1) 
recodlng errors can occur 1n creating the variables (e.g., ranges of 
22-44, 35-54, 55-64) that result 1n overlapping the age ranges so that 
Individuals age 35-44 would appear 1n both of the first two groups 
(I.e., subgroups are not mutually exclusive); and (2) Individuals 1n 
the sample may not fall Into any of the three age categories created 
(I.e., subgroups are not exhaustive). This could occur 1f some 
participants were younger than 22 or older than 64. If the ^ubgr^ups 
are not exhaustive, then all of the participant observations that do 
not fall Into one of the categories would be treated as comparison 
group members, which would result In biased estimates of the net 
Impacts of JTPA for the other subgroups. 

The Interaction analyses described above will Identify the types of 
Individuals who benefit most from JTPA and wnether there 1s a general 
pattern to the variation 1n program effectiveness. At a minimum, we 
recommend that states examine differential Impacts by race, education, 
age, UI claimant status, AFDC recipient status, and pre-program 
earnings. Because Individuals' pre-program characteristics cannot be 
• affected by JTPA, no additional selectivity bias 1s Introduced 1n 
disaggregating JTPA net Impacts by demographic subgroups. 25 However, 



" If the characteristics defining the subgroups of Interest are not 
measured equally well for the participants and comparison group 
members, however, the subgroup Impacts will Inappropriately reflect 
these differences. That 1s, because the presence of measurement error 



i?DA We f d / SC 4. r J be below - this may not be the case when examining whether 
JTPA effectiveness varies by program activity. ■»■■■»« wne.ner 

lrJ^" C l P } e ' 1°. Pr ° be beneath tne avera 9 e net ^pacts of JTPA and 
provide Information on the program activities that contributed to the 
average effects one would perform an Identical Interaction analysis to 
the one described above using , Z 2 , Z 3 , and Z 4 to represent 
classroom training on-the-job-traLngf Job 3 search assistance? and 
2S?Lf P H° gram «; S r S I a 1 Ce reSDe ct^ely. Then, If c , represented the 
III 7 ated c° ef f1c 1ent for the Interaction term between the JTPA dummy 
and Zj, then ci would be the estimate of the average net Impact for 
CT, c 2 would represent the estimated net Impact for OJT, C o would 
represent the estimated net Impact for JSA, and c 4 would represent 
the estimated net Impact for other JTPA activities. However, as we 

tu* K „ b ; B ,°k W ' K there 15 3 K Jo [ Pr0blem that t^eatens tha Internal 
validity of the by-program activity net Impact analysis. 

The problem 1s the familiar one of selection bias. In this context 1t 
relates to the nonrandom assignment of program activities to JTPA 
participants. As described above, the assignment of program activity 
11 ^ I ?, n A baseQ on the agency's perception of an Individual's 
needs and abilities. To the extent that this assignment process 1s 
based solely on the measured characteristics of participants (1 e 
age, race sex, education, pre-program earnings), this will not bias 
the net Impacts by program activity as these characteristics will be 
included 1n the net Impact model. A much more serious problem arises 
Llf foment of program activities 1s based on unmeasured 
ers su c h as motivation and ability, and those unmeasured 

characteristics also affect earnings. if, for example the 
more-motivated participants are assigned to OJT programs and are also 
more likely to have higher earnings, then the estimated coefficient of 

roIffl°?L ma ? nu 96 and posU1ve - However, this significant 
coefficient of OJT on earnings would not measure the true effects of 
? L ?H " rn1n g s - but merely reflect the fact that more-motivated 
IJtlt ^ /! re a " 1 9 ned ^ the OJT activity. Thus, one must be very 
ilfl? 1 ? interpreting net Impacts by program activity because o: th1? 
additional selection bias that can occur as a result of the assignment 
process. 3 

As described 1n the evaluation guide, there are statistical procedures 
that can potentially be used to correct for such sele..ti:,n biases 
These procedures rely on what are known as "1nstrume".al variable" 
techniques. However, such techniques are only useful 1f one -an 
Identify variables to play the role of Instruments. Specifically, one 
must find variables that affect the assignment of program *ct1v1+1es to 



n an Independent variable biases Its estimated coefficient downward, 
j;. 8 amou 4 n K t of measurement error on a subgroup characteristic were 
greater 1n the JTPA sample, for example, than 1n the comparison group, 

l' 6 //" °J. fc ha : characteristic on the outcome variable would be 
smaller 1n the JTPA sample than 1n the ES registrant sample. The 
m n oI a SJ^ n „ term would Inappropriately pick up such a difference aiid 

III lVl q] l 1nd1c ate that JTPA Impacts were smaller for Individuals 
with that characteristic. 
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particular Individuals but that do not affect participants 1 
post-program outcomes. Althcugh this Is a difficult task 1n any 
circumstance, 1t will be particularly difficult given the limited 
number of Individual characteristics available for the proposed 
analysis . 

It may, however, be possible to . identify potential Instrumental 
variables from a carefully structured process analysis. As described 
1n the SDA level A Guide for Process Evaluation (Volume III), an 
Important aspect of the process analysis will be a detailed description 
of the process Involved 1n assigning program activities to specific 
participants. Such an analysis might Indicate, for example, that the 
assignment of program activities 1s Influenced primarily by the 
availability of program-activity slots 1n a given SDA, and since the 
availability of program slots should not affect post-program earnings, 
such a variable could potentially serve as an Instrument. 

Provided one has Identified one or more variables that can serve the 
ro?e of Instruments, the following steps must be followed to correct 
for selection bias using this procedure: 

< Estimate separate regression equations with dummy variables vor 
each program activity as the dependent variable and Include as 
Independent variables all of the variables that appear 1n the 
basic net Impact model (except the JTPA participant dummy) plus 
the instrumental variables. These equations should be estimated 
separately for adult men and women over the entire program year 
sample of participants. 

2. Calculate a predicted probability for each participant of being 
assigned to each of the alternative program activities using the 
estimated coefficients from the regression models estimated 1n 
Step 1. Most analysis software packages have the capability of 
either producing these predictions as part of Step 1, or of 
saving ti,e regression coefficients from the first step and 
allowing tt.e analyst to prepare recode statements to compute the 
predicted values. 

3. Replace che program activity dummy variables in the net Impact 
equation with these predicted values for participants and set 
th*? values to zero for comparison group members. Provided the 
Instrumental variables strongly affect the assignment of program 
activities tc particular individuals but do not affect 
participants post-program outcomes, the coefficients of the 
predicted program activities should produce consistent estimates 
of net impacts by program activity. 



E»vft if the process analysis is not successful 1n identifying specific 
varlab :fs to p:*y the role of instruments, 1t could shed considerable 
Ugh* on the validity of the program-activity net Impact estimates and 
provide some Indication of the likely direction of the selection 
biases. For example, the process analysis should be able to determine 
which urogram activities the more- and less-job ready participants are 
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assigned. 26 Then, if the Impact analysis finds that a given oroaram 
activity has a large positive effect on post-program earnings (negative 
effect on AFDC grants) and the process analysis Indicates that the morp 
disadvantaged Individuals are assijned to that activity, u would 
Increase confidence that the observed relationship 1s causal and not 

?hf<A°h S ! 1 !5 t li n , bl3S - . U L J howev * r ' the P^cess analysis Indicates 
that the individuals assigned to that program activity are considerably 
more advantaged, then one should not be very confident that the net 
Impact estimates for that program activity solely reflect the effects 
of the program. 

OBTAINING NET IMPACT ESTIMATES BY PROGRAM LENGTH OF STAY 

In addition to providing evidence on how the net Impact of JTPA varies 
by Individual characteristics and program activities, 1t may be useful 
to examine whether the Impact of JTPA varies by participants' proqram 
experiences. For example, several studies have attempted to determine 
how the Impacts of employment and training programs vary with length of 
stay 1n the program. Although the results of such analyses could 
;rov1de Important information about the mechanisms through which 
employmef?t and training programs produce their effects, as we describe 
below one must be careful 1n Interpreting the results due to the 
familiar problem of potential selection bias. 

To Investigate whether the net Impacts of JTPA vary by length of stay 
In the program,, one would estimate an autoregressl ve model like those 
described above with the overall program participation dummy variable 
replaced by JTPA dummy variables Interacted with variables developed to 
measure length of program participation. 27 For example, one could 
categorize the length of stay variable Into four dummy variables 
corresponding t< stays of less than one month, one to three months 
three to six months, and greater than six months, and Interact these 
four dummies with the JTPA participation dummy. Then the estimates of 
the four program Interaction coefficients would represent the average 
Impacts of JTPA for Individuals with these different lengths of stay. 

Information obtained from comparing the measured characteristics 
of participants across program activities should also provide some 
information on the probable direction of the selection bias. For more 
detail, see the discussion above of the analysis proposed to determine 
whether the comparison group meets comparability Criterion (1). 

27 One might also consider Including a dummy variable that 
represents whether the participant c ompleted the program activity the 
person was assigned to, as well as an Interaction term between length 
of stay and completion status. Such variables will help to control for 
V • ses - ue t0 measurement error. For example, 1t may help to account 
fur problems that arise by combining enrollees 1n a 
student-learnlng-paced adult basic education class that has an 
extremely variable length of participation with Individuals 1n an 
electronics classroom training program that may have a fixed period of 
training, and representing the J1PA treatment provided by a common 
classroom training dummy variable. 
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Alternatively, 1f the effects of length of stay on the outcomes are 
approximately linear, a convenient specification involves a model with 
a JTPA participation dummy and the participation dummy Interacted with 
actual weeks 1n the program minus average number of weeks In the 
program. In this specification, the coefficient of the JTPA dummy 
represents the estimated Impact of JTPA at the average length of stay, 
and the coefficient of the Interaction term 1s an estimate of the 
dollar Impact of an additional week of program participation. 

Although 1t 1s straightforward to construct the length of stay 
variables and estimate the coefficients of the interaction terms In the 
net Impact models, one must be very careful In Interpreting the results 
because of the familiar problem of selection bias. Although the 
autoregresslve earnings model controls for differences In measured 
characteristics between short- and long-term participants, it Is likely 
that some differences In unmeasured characteristics remain. 
Individuals who leave the program early may be less motivated or, 
alternatively, may have found employment on their own. On the other 
hand, Individuals who stay In the program a long time may do so because 
they have fewer other employment opportunities. Length of stay Is also 
11kel> to depend on the type of program activity and SOA 
characteristics. 

Because of these additional selection bias problems, one should use 
extreme caution In Interpreting the estimated Impacts by length of stay 
as representing causal relationships. In order to overcome these 
biases and obtain estimates of the net Impacts of JTPA by length of 
stay that one has confidence In, It is necessary to statistically 
adjust for the selection process. One method would Involve using 
Instrumental variable techniques. Operationally, as described above, 
one would first estimate a regression equation to obtain a predicted 
value of length of stay. Such equations would be estimated using the 
participant sample only, and would Include as Independent variables all 
variables In the net Impact mode; plus the Instrumental variables. The 
predicted values would then be entered In the net Impact equation for 
participants, and zeros would be Included for comparison group members. 

The success of such an Instrumental variable procedure relies heavily 
on the ability to Identify variables to play the role of Instruments. 
That Is, one must find variables that are highly correlated with length 
of stay, but that do not Influence earnings or AFOC grants. Depending 
on the types of Information obtained from participants and Included In 
the JTPA HIS In different states, It may be possible to Identify 
variables than can play the role of Instruments for length of stay. 
For example, It may be possible to develop Instruments from answers to 
very simple questions about participants' general satisfaction with 
JTPA services and whether participants were assigned to the type of 
program they wanted. That Is, It seems plausible to assume that 
Individuals vho are satisfied with the program, or who are assigned to 
the program type they were most Interested In, would remain In JTPA 
longer. At the same time, however, there Is no obvious reaso.i why 
satisfaction with the program or assignment to the desired training 
program should affect earnings or AFOC grants, Independent of the 
effect of the training and the actual length of stay. Thus, these 
variables could be entered Into a regression equation along w11>\ other 
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demographic characteristics to explain program length of stay and then 
a predicted value for length of stay could be constructed and entered 
in the net Impact equation 1n place of the actual length of stay. 

For states that are very Interested 1n overcoming these selection bias 
problems, they should carefully review the data 1n their JTPA MIS as 
well as the Information obtained during the process analysis to see 1f 

^ M f?I? S J that affect length of sta *> but not earnings, can be 
Identified. If successful, they should Implement the Instrumental 
variable approach and determine whether the variables selected 1n fact 
strongly affected length of stay. If not, then 1t 1s net necessary to 
go to the second stage of Including the predicted value 1n the net 
Impact model, as the predicted variable will be too highly correlated 
with other characteristics 1n the model, and the results w'lll not be 
reliable. If, however, the Instruments do strongly affect program 
length ot stay, then the net Impact model should be estimated with 
predicted length of stay replacing the actual value If the 
Instruments have been successful, then they will purge the correlation 
between the error term and length of stay, and result 1n consistent net 
Impact estimates. 

ADJUSTMENTS FOR POTENTIAL DATA AND DESIGN DEFICIENCIES 

The models described above Involve a comparison of the Ul Wage Records 
and AFDC Grants Records of JTPA participants with those of comparison 
groups derived from ES registrants. In addition to the problem of 
potential selection bias, there are some deficiencies 1n the UI 
earnings data and 1n the JTPA and ES data that may affect the results 
For example, UI earnings are Inadequate 1n that they do not reflect 
earnings from Jobs that are not 1n covered employment, or earnings from 
Jobs that are located across the border 1n other states. The JTPA and 
ES data are deficient because there 1s Inadequate Information on 
whether ES registrants participated 1n JTPA, which may result 1n a 
contaminated" comparison group. In this section, we briefly discuss 
the likely extent to which the basic Impact estimates will be affected 
by these data and design deficiencies and the types of adjustments that 
may be necessary. 

In Chapter 4, we outlined procedures that co"ld be used to reliably 
exclude Individuals from the comparison group who were currently 
participating 1n JTPA, who had participated 1n JTPA 1n the previous 
year or who participated 1n JTPA during the post-program periods being 
examined. If, however, for various reasons 1t 1s not possible to 
Implement these procedures, the comparison group may be "contaminated" 
to a certain extent. Such contamination would lead to an underestimate 
of the net Impacts of JTPA, since H would effectively dilute the 
treatment, as some comparison group members would have also received 
JTPA services. We do not believe, however, that the net Impact 
estimates will be seriously affected by this potential contamination 
problem, as the magnitude of the Droblem \s likely to be smaller than 
one might expect. That 1s, although the ES 1s one source of applicants 
for JTPA programs and, as such, one might expect that contamination 
could be high, existing data Indicate otherwise. For example, based on 
data for the State of Washington for PY 1985, only 0.]% of all ES 
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registrants active during the year were recorded as having enrolled In 
JTPA programs. Moreover, only 0.3% of those economically disadvantaged 
enrolled 1n JTPA. Although the figures are somewhat higher for 
enrollment 1n any training program (e.g., JTPA, Job Corps, WIN, other) 
--1.0% for all applicants and 3.154 for those economically disadvantaged 
—even these participation rates are small enough to be safely Ignored. 

In states that have higher probabilities of economically disadvantaged 
ES registrants enrolling 1n JTPA, and 1n which 1t Is not possible to 
exclude those who participate 1n JTPA from the comparison group before 
conducting the net Impact analysis, It may be necessary to make some 
aggregate adjustment to the net Impact estimates. Specifically, 1f p 
(q) 1s an estimate of the proportion of the adult men (women) In the 
comparison group participating In JTPA during the period of enrollment 
through the post-program period (I.e., from one to two years), the 
estimated average program net Impacts for adult men (women) should be 
mult ;ed by 1/(1 -p) (l/(l-q)) to adjust for this problem. Although 
It Is difficult to obtain precise estimates of p and q, we believe that 
a reasonable approach would use values 1n the range of .05 to .10. 

The second major data deficiency concerns the fact that UI Wage Records 
do not Include jobs In nonreportlng employment, nor do they Include 
earnings from jobs 1n other states. As discussed In the evaluation 
guide, because the net Impacts are based on differences 1n earnings 
between participants and comparison group members, the omission of 
earnings due to these problems biases the estimated Impacts of JTPA 
only 1f program participation affects the probability of working 1n 
uncovered employment or the likelihood of working 1n another state. 
Although this was an Important concern 1n evaluating the net Impact of 
CETA programs given the high likelihood of CETA PSE participants 
turning their training slots Into subsequent jobs 1n the public sector 
(which are less likely to be In reported employment), given the focus 
of J1PA on employment In the private sector, this should be less of a 
problem for the state-level net Impact model. Moreover, 1n order to 
create a meaningful adjustment one would need Information on 
Interview-reported earnings and UI earnings for both groups 1n the 
post-program period, which will not generally be available. Thus, the 
best one can do 1s acknowledge the potential problem and Indicate that 
the net Impact estimates are based on the reasonable assumption that 
J1PA does not affect the probability of working In nonreported 
employment or working across the border 1n other states. 

Finally, the most Important potential adjustment to be considered Is 
for selection bias, that 1s, systematic differences between 
participants and comparison group members that cannot be directly 
controlled for 1n the autoregressl ve net Impact model. One can attempt 
to adjust for selection bias 1n two ways: (1) use evidence on 
pre-program differences between the two groups, or (2) estimate other 
more complex statistical models that under certain assumptions can 
correct for differences In unmeasured characteristics between the two 
groups. Using the first approach, one would use estimates of 
differences 1n adjusted pre-program earnings or AFOC grants between 
participants and comparison group members that are due to unmeasured 
characteristics that are developed as part of the analysis to examine 
the adequacy of the comparison groups described earlier In this 
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section. The size of the estimated difference 1n pre-program earnlnos 
2S„-f° K Unm l a , SU K red characteristics 1s a reasonable estimate of the 
amount by which earnings Impacts could be overstated or understated 
(depending on whether It 1s positive or negative) 1f the difference 
persisted in the post-program period. Thus, for example. If adul men 
women) JTPA participants are estimated to have earned $100 more ($200 
less) in the Immediate pre-program year than Individuals In the 
ZTrl 1s °" groups using this first approach one would adjust the main 
Impact estimate (I.e., the coefficient of the JTPA dummy variable) 

h 0W !!w^ l U J"? r ? > by * 100 ($200) for men ( women >- » should be noted 
however, that because pre-program earnings are controlled for 1n the 
autoregresslve net Impact model, this adjustment may overcompensate for 
the selection bias due to differences 1n unmeasured characteristics 
between the two groups. 

The second approach to correcting for selection bias Involves using 
somewhat more complex statistical models to estimate the net Impacts of 
JTPA. For example, as described 1n the evaluation guide 
symmetric-difference'' regression models can overcome problems due to 
differences 1n unmeasured characteristics between the groups and yield 
U S a ! e l n6t 1mpact est1mates Provided certain assumptions are met It 
should be noted, however, that such models require considerably qreater 
longitudinal data-at least three to five years of UI Wage Records and 
S , Reco [J s :- wh1ch will limit their usefulness 1n certain 
states states that are Interested 1n estimating such models to 
correct for selection bias should consult the evaluation guide. 
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CHAPTER 6. 



COST ANALYSIS AND BENEFIT-COST COMPARISONS 



The previous chapters of this Implementation guide have described how 
to estimate the net Impacts of JTPA on participants 1 post-program labor 
market experiences. Results of these analyses could be used to 
estimate the benefits of JTPA to participants and, under certain 
assumptions, to estimate the benefits of J 7 PA to society as a whole. 
In order to determine whether JTPA 1s an efficient use of public 
resources, however, 1t 1s necessary to assess the costs of providing 
JTPA services, and compare the costs and benefits c? the program. 
Moreover, to make Informed decisions about the design of the program, 
policy-makers must know both the costs and benefits of specific program 
activities. That is, program activities that yield relatively small 
benefits may yet be very effective when compared to the costs 
Involved. In this chapter we provide a general framework for comparing 
the costs and benefits of JTPA and describe specific :r 0 cedures for 
calculating the benefits and costs of JTPA. 

The purpose of a benefit-cost analysis is to judge the s ocial 
efficiency of a program. As such, it attempts to determine whether the 
value of the goods and services available to society is greater as a 
result of the program or whether the value would have been even greater 
had the resources been put to alternative uses. In order to make this 
determination, all of the benefits of the program are assigned a 
monetary value, and the present value of the benefits 1s compared to 
the present value of the program costs. Assuming that all present and 
future benefits and costs are identified, that appropriate monetary 
values are assigned, and that an appropriate Interest rate is used to 
discount future benefits and costs Into current values, then JTPA would 
be Judged to be a worthwhile use of public resources if the present 
value of the benefits is larger than the present value of the costs. 

Although the process Involved in conducting a benefit-cost analysis is 
conceptually straightforward, there 1s, in practice, a variety of 
Issues that limit the validity of any benefit-cost analysis. As a 
result, because of data limitations and other Issues described 1n 
detail in the evaluation guide, it will not be possible for states to 
conduct a comprehensive benefit-cost analysis to provide a definitive 
estimate of the return on the Investment to JTPA. Nevertheless, the 
general approach described below should be a useful tool to st-tes 1n 
organizing information on program benefits and costs and should enable 
them to get an overall sense of the effectiveness of JTPA and the 
conditions under which JTPA can be regarded as an efficient use of 
pubMc resources. 

The discussion below begins with a brief description of a benefit-cost 
framework for analyzing the effectiveness of JTPA. We then briefly 
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describe how to use the net Impact estimates to measure some of the 
Important social benefits due to JTPA, We then describe how to conduct 
a cost analysis to estimate the marginal cost of serving additional 
JTPA participants and the marginal costs of different program 
activities. Finally, we discuss the Issue of discounting future 
benefits and costs so that the benef '» t-cest comparisons can be made 1n 
constant dollar terms and describe a few additional comparisons that 
should be made to determine how sensitive the overall conclusions are 
to certain assumptions. 



BENEFIT-COST FRAMEWORK 

The benefit-cost framework presented 1n Exhibits 5 and 6 r respectively, 
lists the major benefits and costs that would Ideally be accounted for 
1n conducting a comprehensive benefit-cost analysis of JTPA. As an 
aide to keeping track of the different benefits and costs, they are 
presented from three perspectives: the participant, the taxpayer, and 
society as a whole. The first class of benefits and costs consists of 
those that are received by or borne by program partlcloants . The 
participant perspective 1s Important because 1t sheds light on 
Individuals 5 Incentives to participate 1n the program. The taxpayer 
perspective, sometimes referred to as the nonpartlc Ipant perspective, 
1s Important because It reflects the effects of the program on the 
government budget. Finally, the most inclusive set of program benefits 
and costs are those accruing to society as a whole. These are simply 
the sum of benefits and costs received or borne by participants and 
taxpayers, taken separately. As such, the social perspective Ignores 
transfers between segments of society (I.e., participants and 
taxpayers) and examines whether the program results In a net Increase 
1n the resources available to society. 28 This Is the appropriate 
perspective for examining the overall effectiveness of JTPA. 



Benefits 

As Indicated In Exhibit 5, the major benefit of JTPA from the social 
perspective 1s the Increase 1n output produced by participants. 
Conceptually, two types of output gains should be distinguished: (1) 
the Increase 1n post-program output, and (2) the Increase 1n output 
produced while participate s ' the program. For the most part, the 
1n-program output due to JTPA 5 likely to be small, particularly for 
participants 1n classroom t 1, ng and job search assistance program 
activities. Only for OJT |?i ^grains Is the value of 1n- program output 
likely to be substantial, and even for these programs, 1t Is difficult 
f;o assign appropriate monetary values without an extensive data 
collection and analysis effort. Because of the difficulties Involved 



*° Reductions in transfer payments (e.g. , AFDC grants) do not 
represent a benefit from the social perspective since the Increased 
benefit to taxpayers 1s offset by the loss of Income to recipients, and 
there Is therefore no change In the resources available to society as a 
whole. Thus, Including estimated benefits from reductions In welfare 
dependency due to JTPA would Involve a double counting of benefits. 
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1n measuring the value of 1n-program output, a> well as 1n measuring 
the value of other nonmonetary benefits (e.g. , reduced crime) , we 
recommend that states and SDAs do not attempt to directly measure these 
benefits, but recognize their potential Importance when discussing the 
overall results from the benefit-cost analysis. As n result, the 
primary benefit to be measured Is the Increased post-program output due 
to JTPA. 

As we describe 1n the evaluation guide, to measure the Increase 1n 
output of goods and services available to society 1n the post-program 
period due to JTPA, we recommend that the participant-comparison group 
differences 1n earnings 1n the post-program period be used. This 1s a 
reasonable procedure provided that JTPA participants do not find jobs 
1n the post-program period at the expense of other disadvantaged 
persons (I.e., program participants do not displace other job 
seekers). 29 It 1s beyond the scope of the state-level model to 
assess the extent to which Individuals who participated 1n JTPA 
displaced other job seekers As a result, the benefit-cost analysis 1s 
limited to determining wnetner the benefits to participants from 
receiving JTPA services are greater than the costs to society of 
providing those services. 

In translating participant earnings gains Into a measure of the 
Increase 1n output of goods and services available to society, two 
additional Issues arise. First, one must :?rm1ne how to extrapolate 
the post-program gains observed for the perils from three months to 
one year following termination Into subsequent periods. For example, 
1f the three-month, six-month, and one-year earnings Impacts Imply 
similar gains per quarter, then 1t may be reasonable to assume that the 
gains persist over time. However, based on previous studies, it 1s 
likely that the gains decline over time, and Information through just 
the first post-program year may not be sufficient to estimate reliable 
time trends for the purpose of extrapolating future gains. Because of 
this ambiguity, the benefit-cost analysis should Indicate whether 
earnings gains observed during the one-year follow-up period are 
sufficient to make the program worthwhile, as well as Indicate the 
assumptions about future benefits that are necessary to make JTPA 
viewed as a worthwhile Investment. 

A second Issue concerns adjustments that should be made to earnings 
gains to account for fringe benefits. That 1s, 1f the Increase 1n 
output 1s equal to the Increase 1n compensation paid to those who 
participate 1n JTPA, then although this compensation is primarily 1n 
the form of monetary earnings, adjustments for nonmonetary earnings 
should also be made. Fringe benefits Include pensions, health and 
other forms of Insurance, and payments on behalf of the worker for 
unemployment Insurance and worker compensation, and FICA. As a rough 
approximation, Insurance and pension benefits for workers served by 
JTPA are estimated to be approximately 10% of monetary benefits, and 



Although very unlikely, 1n the extreme the program could produce 
no net Increase 1n output despite large Increases 1n participants 1 
post-program earnings by simply reshuffling jobs from nonpartldpants 
to participants. 
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payments to government programs are approximately 10% as well. Thus, 
we recommend that the net earnings gains be multiplied by 1.2 to adjust 
for fringe benefits 1n deriving a measure of the social benefits due to 
JTPA. 

To summarize, the benefits to be measured and Included 1n the 
benefit-cost analysis Include only the Increase 1n post-program output 
due to JTPA. This will be approximated by the Increase 1n post-program 
earnings, as measured by the estimated coefficient of the JTPA dummy 
variable 1n the net Impact equation (and potentially adjusted for data 
and research design limitations as described 1n Chapter 5) f adjusted 
for fringe benefits. Procedures will be developed to determine how the 
Increase 1n earnings over the first year should be extrapolated to 
yield estimates of Increases 1n post-program output 1n subsequent 
years. These steps will yield an estimated stream of future benefits 
for both adult men and women. Individual values In these benefit 
streams will then be weighted by the proportion of men and women served 
by JTPA in the particular program year to generate an estimate of the 
aggregate benefit stream due to JTPA. 



Costs 

As indicated in Exhibit 5, there are several different cost components 
1n a benefit-cost analysis of an employment and training program. The 
major cost categories Include (1) program operating costs; (2) 
participant opportunity costs; (3) transfers to participants; (4) costs 
associated with participating in the program activities; (5) psychic 
costs to participants of participating in JTPA; and (6) stipends and 
other income maintenance transfers such as UI benefits received during 
training. Each of these cost categories is discussed briefly below. 

The major costs from the social perspective are the program operating 
costs. The costs of operating employment and training programs Include 
direct operating costs such as premise rent, salaries for instructors, 
and costs of materials and supplies, and Indirect costs such as the 
costs of managing and administering the program. As Indicated 1n 
Exhibit 5, because program participation 1s free, the operating costs 
are not considered as costs from the perspective of program 
participants. However, operating and administrative costs do involve 
the use of considerable resources that have alternative uses and, as 
such, they represent real costs from the perspective of the taxpayer 
and society as a whole. Thus, 1n Exhibit 5, these costs are 
represented as a zero to program participants and as a minus in the 
other two columns. 

A second Important component of cost concerns the earnings 
opportunities that participants may forego while participating 1n the 
program. These foregone earnings are clearly costs -:o participants 
and, to the extent that less output 1s produced because workers were 
participating 1n JTPA, the foregone output (as measured by foregone 
earnings) 1s a net cost to society as wall. Although prsvlous studies 
have recognized foregone earnings to be an important element of program 
cost, because of data limitations and certain evaluation issues this 
cost component is almost always excluded from the final benefit-cost 




Exhibit 5 



A GENERAL TAXONOMY OF THE BENEFITS AND COSTS 
OF JTPA FROM DIFFERENT PERSPECTIVES 



Perspective 



Participant Taxpayer 



Benefits 



Costs 



Increased output 

Post-program output + 0 



In-program output 0 



Nonmonetary benefits 
ft educed crime 



Reduced receipt of transfers 

Reduced welfare recipiency + 
Increased taxes - + 



Reduced use of alternative 0 + 

social programs 
Improved work attl^ides of + 

participants 



• Program operating costs 0 
(e.g., premise costs, staff costs, 
materials and supplies, and 
administrative costs) 

« Participant opportunity costs - 0 

(e.g., foregone earnings) 

• Transfers to participants + 
(e.g., stipends) 

• Costs of participation 0 
(e.g., transportation, child care) 

• Psychic costs - 0 
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comparisons. Although the social program costs may be somewhat 
understated by this exclusion, given the relatively brief length of 
participation In JTPA and the questionable nature of the assumption 
that considerable output was foregone when previously unemployed 
workers participated 1n the program, we recommend that this cost 
component be formally omitted from the benefit-cost analysis. However, 
1n Interpreting the results of the benefit-cost comparisons, one should 
Indicate to what extent the overall assessment Is likely to be 
sensitive to omitting the social cost of foregone earnings, 30 

Other potentially Important costs of JTPA from the taxpayers' 
perspective are the transfers to participants In the form of stipends 
and other Income transfer payments, and transportation and chlldcare 
allowances. Although such costs are much less Important under JTPA 
than under CETA, they could be considerable 1n some cases. It should 
be noted, however, that such transfers are a cost from the taxpayers 1 
perspective, and a benefit to the participants that receive them and, 
as such, they do not affect the social cost of the program. 

Other potentially Important costs concern the direct cost participants 
Incur In participating In JTPA activities as well as any psychic 
costs. The psychic costs are Inherently unmeasurable and are Included 
In the conceptual framework only for the purpose of completeness. 
Moreover, although the costs Incurred by participants In attending 
classes or participating In job search activities that they would not 
h?ve done if they had not participated (e.g., Increased transportation, 
chlldcare, special clothing/equipment) are potentially measurable, they 
require data from Individual participants. Because of the large 
expense Involved In acquiring such Information, and given that these 
costs are likely to be a small share of the total cost of the program 
to the Individual and to society as a whole, as a practical matter 
these costs will also be omitted from the final benefit-cost 
comparisons. 

Thus, to summarize, the costs of JTPA to be measured and Included In 
the benefit-cost analysis will be limited to the costs Involved In 
operating the program (I.e., the sum of premise rent costs, staff 
costs, materials and supplies, and administrative costs). In 
interpreting the benefit-cost analysis comparisons, however, 1t will be 
Important to recognize that many of the social costs of JTPA have not 
been measured and that these unmeasured costs could affect the overall 
assessment of whether JTPA 1s an efficient use of social resources. 
Below we describe an approach to estimating the operating costs of JTPA. 



JU For example, one could obtain a gross estimate of the foregone 
earnings from data on the earnings of comparison group members during 
the program period . Specifically, one could use the earnings of 
comparison group members for the quarter of enrollment pins half of 
their earnings for the subsequent quarter to approximate foregone 
earnings over a four-and-a-half month period, roughly the average 
length of stay In JTPA. In interpreting the final benefit-cost 
results, one could then indicate whether adjusting for this gross 
estimate of foregone earnings affects the overall conclusions. 




ESTIMATING PROGRAM COSTS 



Because the benefits from JTPA will be expressed In terms of dollars 
per Individual adult participant, the cost analysis must similarly 
estimate the Incremental (I.e., marginal) cost 1n dollar terms per 
Individual adult participant. As descrlDed In the evaluation guide, 
although there are many obstacles to deriving reliable estimates of the 
marginal costs of serving JTPA participants (e.g., problems of data 
omission and Inconsistent aggregation, difficulties In allocating Input 
costs among Joint outputs, ambiguity Involved In Imputing prices of 
owned or subsidized Inputs), statistical methods are available that can 
overcome several of these problems and provide useful Information on 
the marginal costs of employment and training programs. Below we 
describe one method that states could use to estimate the marginal cost 
of serving adults In JTPA. States that are interested In alternative 
methods or ways of estimating program costs at the SDA level should 
consult the evaluation guide. 

The primary source of data for the cost analysis will be the J1PA 
Annual Status Report (JASR). The JASR provides for each SDA the 
characteristics of program termlnees, and Information on program 
outcomes and costs, for Title II-A and Title III programs funded under 
JTPA. 31 Fortunately, for our purposes, these data are provided 
separately for adult and youth participants in TUle II A programs. 
Moreover, because these data are structured for reporting to the 
federal government, they are consistently available for all SOAs and 
the quality of the information appears to be very good. 

As described It the evaluation guide, these data have several 
advantages for estimating the marginal cost of serving JTPA 
participants. In addition to being in a standardized format with 
unambiguous definitions of all Information items, the JASR contains 
Information on total federal expenditures in operating JTPA (although 
not tot*: social costs because it Includes stipends and excludes 
opportunity costs and other costs of program participation), as well as 
considerable Information on the socioeconomic characteristics of adult 
program termlnees (e.g., number of termlriees by sex, age, education, 
racial/ethnic group, welfare recipient status, limited English language 
proficiency, handicapped status, and average number of weeks 
participated). These participant characteristics can be thought of as 
Inputs that enter the employment and training production process and 
that have obvious Instructional and resource implications that will 
affect costs (e.g., those participants with limited English language 
proficiency will likely require more program resources to complete 
training). As such, these characteristics can be used to standardize 
the relationship between total costs and participants served to obtain 
estimates of marginal costs as described below. 

Although the JASR contains considerable information and could serve as 
the basis of a cost analysis, it also contains two major limitations. 



Although the cost Information corresponds to program termlnees 
and the benefit Information Is based on net Impact estimates for 
samples of program enrollees, this should not cause any comparability 
problems as long as program costs do not vary widely from year to year. 
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Specifically, the JASR does not contain (1) data on the number of 
termlnees by type of program activity or (2) Information on the costs 
incurred at the state level in operating JTPA programs. Without 
Information on the number of Individuals served by type of program 
activity. It Is not possible to Isolate how the marginal costs of JTPA 
activities differ, which would in turn preclude one from comparing the 
marginal benefits and marginal costs of different program activities to 
Identify those activities that are most effective. In addition, 
without Information on the costs Incurred at the state level, the 
marginal costs of serving JTPA participants as derived only from JASR 
data would be understated. 

In order to overcome these problems, we offer two recommendations. 
First, and most important, every effort should be made to obtain data 
on the number of adult termlnees that participated In various program 
activities during the program year. At a minimum, it would be useful 
to have Information on the number of classroom-training-only termlnees, 
the number of OJT-only termlnees, the number of JSA-only termlnees, the 
number of termlnees who only participated In some other activity, and 
the number of termlnees who participated in multiple activities. In 
order to be useful for the cost analysis, this information would have 
to be obtained from each SDA, and for the same period In which the 
termlnee characteristics and program costs on the JASR are reported. 
This could be accomplished either by adding the data elements to the 
JASR form and requesting SDAs to provide the information, or by 
developing the necessary computer software to extract the Information 
from the JTPA MIS. For example, one could implement the following 
steps to obtain the necessary Information for PY 1985. 

1. Create a working file (on tape) of all persons who terminated 
from JTPA Title II- A programs In any SDA in the state during PY 
1985. The file should Include the person's age, data on all 
program activities participated In, and an SDA identifier. 

P. Exclude from the file all oersons who are 21 or younger. 

3. Create variables that represent each type of treatment of 
Interest, and that may have di f ferent cost structures . For 

"example, as indicated above, it Is likely to be Important to at 
least differentiate the costs by type of activity as well as 
costs for those who participate In only one activity versus 
multiple activities. T!iis can be accomplished by creating five 
variables, the first four of which would simply be dummy 
variables Indicating whether the only activity the person 
participated in was CT, OJT, JSA, or other, and a fifth 
variable indicating whether the person participated In any 
combination of these activities. 

4. Stratify the adult termlnee file by SDA. That is, create a 
separate subfile of adult termlnees for each SDA. 

5. Create counts of the number of individuals In each SDA in each 
of the five program treatment types and merge these counts with 
the JASR data for each SDA. 
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Our second recommendation concerns how to handle costs incurred at the 
state level in the operation of JTPA programs. Conceptually,, the costs, 
(actual or imputed) incurred at the state level in the operation of 
JTPA programs should be added to annual program year SDA total costs ta 
obtain a better estimate of the total social costs of JTPA. Provided 
information is available on the total costs contributed by the state to 
the operation of JTPA, it would be possible to apportion these costs to 
the various SDAs. For example, one method would be to assume that 
ccrsts incurred at the state level tr< support of various SDAs are 
proportional to the number of adult termlnees in ths SDA. Thus> to 
allocate state-level costs in operating JTPA programs to the different 
SDAs, one could multiply total state costs by the ratio of the number 
of aJult termlnees in a given SDA to the total number of adult 
termlnees In all SDAs. Such a procedure would, in part, overcome the 
limitation of the JASR data described above. If, however,, it is not 
possible to obtain estimates o c costs contributed at the state-level to 
the operation of locaT JTPA programs, this limitation would also have 
to be recognized in interpreting the results of the benefit-cost 
analysts. 

With the basic data set and adjustments described above, it Is possible 
to estimate a program cost function that can provide information on the 
marginal cost of serving J1PA participants for the benefit-cost 
analysis. For example, using ordinary least squares regression 
techniques,, one could estimate a regression equation with total federal 
expenditures plus allocated costs incurred at the state level (if 
possible) for the SDA as the dependent variable, and independent 
variables would include: 

• Number of adult termlnees; 

• Number of adult termlnees who are: 

- Hale; 
Black; 
Hispanic; 
Other nonwhite; 

Students (high school or less); 
High 1 school graduates; 

- Age 22-54; 
Welfare recipients; 

Single household heads with dependent children; 
UI claimants; 

Limited English language proficiency; 
Handicapped; and, 

• Average number of weeks of participation. 



With these independent variables in the regression equation, the 
coefficient of the variable "number of adult terminees" would represent 
the marginal cost of serving additional adult participants in JTPA on 
average, and the coefficients of the other variables in the model would 
capture the extent to which the marginal cost varied for serving 
persons with specific characteristics. 
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If the procedures outlined above are followed so that data on the 
numbers of termlnees by program activity are obtained for each SDA, 
then oi.c would estimate a second regression equation like the one above 
except that the "number of adult termlnees" variable would be replaced 
by the following four variables: the number of classroom-training-only 
termlnees, the number of OJT-only termlnees, the number of job-search- 
assistance-only termlnees, and the nimber of termlnees that participate 
In multiple activities. In this formulation, the coefficients of these 
four variables would represent estimates of the marginal cost for each 
of the different types of program activities (CT only, OJT only, JSA 
only, or multiple activities). These estimates of the marginal costs 
of serving adult JTPA participants (either overall or by program 
activity) would be compared to the marginal benefits from the program 
In terms of Increased post-program output (either overall or by program 
activity) to make statements about whether JTPA Is an effective use of 
public resources as descrloed below. 

Before discussing the benefit-cost comparisons to be made, one 
additional potential limitation of the cost analysis should be noted. 
This concerns the Isrue of sample size. That Is, since the analysis Is 
based on SDA-level data, the number of observations available In an 
annual cross sectional analysis equals the number of SDAs In the 
state. In relatively small states with few SDAs, there may be 
Insufficient degrees of freedom to estimate a cost equation like the 
one given above and obtain reliable estimates of program marginal 
costs. In such cases one should consider using quarterly JASR data on 
each SDA (and including dummy variables for different quarters to 
account for seasonal cost differences and costs that may occur at the 
end of the contract year) , or other less formal methods that are 
described In the evaluation guide. 



BENEFIT-COST COMPARISONS 

In conducting a benefit-cost analysis, three data elements are 
required: estimates of the benefit stream over time, estimates of 
program costs over time, and the Interest rate used to discount future 
benefits and costs Into current dollars. In the discussion above we 
have described how to develop monetary measures of the benefits and 
costs of JTPA. Below we briefly Indicate how to discount the future 
benefit stream so that the present value of benefits can be compared to 
the current program costs and Indicate the criteria to be used to 
measure the net effectiveness of JTPA as a social Investment. 32 The 
chapter concludes with some examples of other comparisons that should 
be made to determine how sensitive the overall conclusions are to 
alternative assumptions. 

As discussed In the evaluation guide, because the benefits of 
employment and training programs occur over time, It Is important to 



It should be noted that In most benefit-cost analyses, costs are 
also Incurred in future periods and require discounting as well . 
However, given the short-term nature of the JTPA program, all costs are 
Incurred 1n the current period so that no discounting Is necessary. 
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translate this stream Into a common reference period. Conventionally, 
this has Involved discounting future dollars Into present terms, using 
an Interest rate that approximates the alternative costs of the funds 
invested. 

Choice of an Interest rate to be used In discounting Is not as easy as 
1t would seem at first blush. There are competing theories as to how 
to select the correct Interest rate. It Is not necessary to discuss 
these here except to note that the rate will differ for society as a 
whole compared to, say, a state government or an SOA. The rate for 
society as a whole Is net of any effect due to Inflation, while this 1s 
not the case for a local government or agency. The most practical 
guidance on the choice of Interest rats 1s to select that rate at which 
the local government or agency can lend Its funds. The accounting 
office of a local government or agency can provide that figure. 
Currently, lending rates are In the neighborhood of 10%, which Is a 
relatively high number. 

Using a 10% discount rate, one would measure the net effectiveness of 
JTPA by calculating a benefit-cost ratio, where the numerator Is the 
present value of the Incremental benefits due to the program (I.e., 

£j B(t)/(1 .10)*) , the denominator Is the present value of the costs 

Incurred (which require no discounting since all costs are Incurred In 
the current period), "t" refers to the post-program years In which 
benefits are realized, and H n H Is the last year In which benefits are 
realized. Within the numerous limitations described earlier, JTPA 
would be regarded as an efficient use of public resources whenever the 
benefit-cost ratio exceeded 1. Provided one was able to derive 
estimates of the benefits and costs of each of the various program 
activities, similar calculations could be made to determine which 
program activities had larger benefit-cost ratios and could be 
considered to be more cost-effective. 

In addition to obtaining the main benefit-cost results described above, 
which are based on the "best" estimates of the benefit stream, current 
program operating costs, and a 10% discount rate, we believe It Is 
Important that benefit-cost ratios be calculated to demonstrate how 
sensitive the conclusions are to alternative assumptions. In 
particular, alternative benefit-cost ratios should be calculated for. 

1. One additional discount rate, say 3%. 

2. Benefit estimates that do not include adjustments for selection 
bias or for potential contamination. Since each set of 
estimates rests on a different set of assumptions that are 
inherently untestable, it is important to know how sensitive the 
overall conclusions are to the size of these adjustments. 

3. A range of program benefits and costs that reflect the fact that 
the main estimates are subject to statistical imprecision. For 
example, one could construct an upper and lower bound of a 95% 
confidence Interval for the net impact of JTPA on post-program 
earnings by adding and subtracting 1.96 multiplied by the 
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standard error of the JTPA dummy variable to the estimated JTPA 
coefficient. By adjusting both the upper and lower bounds for 
the fringe benefits, one would then obtain an estimate of the 
upper and lower bounds for the Increase 1n post-program output 
due to JTPA. Upper and lower bounds for the marginal cost of 
JTPA can also be obtained by creating a 95% confidence Interval 
around the appropriate regression coefficient (I.e., adding and 
subtracting 1 .96 multiplied by the standard error of the 
estimated coefficient of the number of adult termlnees 1n the 
cost equation). Then, by choosing different combinations (e.g., 
upper bound for benefits and lower bound for costs, lower bound 
for benefits and upper bound for costs), one can provide useful 
Information on how sensitive the benefit-cost ratios are to 
alternative assumptions. 

These sensitivity analyses, 1n combination with the main benefit-cost 
results, should provide a wealth of information on the effectiveness of 
JTPA programs. 
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CHAPTER 7 

RELATIONSHIP AMONG MODELS: NET IMPACT, 
GROSS IMPACT, AND PROCESS ANALYSIS MODELS 
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CHAPTER 7. RELATIONSHIP AMONG MODELS: NET IMPACT. 
GROSS IMPACT. AND PROCESS ANALYSIS MODELS 



desi d an S p C H Mb t d n V^h PreV J°J! S cha P ters . the net impact model has been 
designed to provide states with important information on the 

fill V m n e r SS .f tbe1 / JTPA P r °9 rams - Although much could be learnt 
in^rr^lfn ^ 1 y £\ net 1mpact model » considerable additional 
information could be obtained by also implementing the process and 
gross Impact evaluations described in other volumes in this series 
111-, \ \. S im P° rtant t0 recognize that the net impact model 
complements the gross impact and process analysis models by providing 

aJrZTtllZ J S usef ? t0 both models and b * draw1n 9 on ^formation 
k ! th6 l e m0dels - In th1s concluding chapter, we briefly 
hoDn h i° W 5 he * net 1mpact model relates t0 the ot her models that have 
been developed to assist states and SOAs in better understanding the 
operations and impacts of their JTPA programs. 

fh o S ?\ P < r0CeS ? an , al y s1s 1s a ver Y Important source of information for 
the net impact model. In fact, because of the inherent limitations of 

w a ,r neXper1men ^ al a PP roach to be used to estimate program net 
Impacts, we regard an SOA process analysis as a necessary first step to 
a valid net impact analysis. That is, because the validity of the net 
™n^/ eSUltS reSt ^ ° n the s1m1la nty of the participant and 
Dar??c LfinnT^ f S f leCted ' 1t 15 cr1t 1cal to understand the JTPA 
participation selection process, the factors that govern the assignment 

In^llrlV^n V° Pr ° 9ram f^^lties, and difference: in the content 
and recording of program activities across SDAs. without an in-depth 
process analysis, one would be much more likely to obtain biased 
£f fl. 14^ i M !3 00 1nformat 1on concerning the direction and magnitude 
of the likely biases which could result in serious misinterpretations 
SL,4,L ne J l mPaCt f1nd1n 9 s - For example, as we have indicated in 
i C a f l'. 3 process analysis will provide the following types 
of useful information for the net impact model: 

* ^/wnfTl' 16 a d «ta1led description of the criteria (explicit 
and implicit) used by SOAs and their subcontractors in screening 

i« < applicants to choose individuals for program 
1pat1or \ * s such . the process analysis will yield 
important insights into the extent/type of "creaming" that 
occurs and the likely differences that may exist between 
participants and comparison group members that are not possible 

^nncT^hl V ^ 1m P aCt ""Odel. This Will be Of 

considerable help in interpreting the net impact findings. 

• The process analysis will also include a detailed description of 
^fW4?! CedUr , eS followed 1" assigning participants to program 
activities (e.g., whether more advantaged participants are 
assigned to specific program activities, whether all 
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participants are first assigned to JSA and only those who are 
not Immediately placed are subsequently assigned to CT or OJT). 
This Information will be useful 1n Interpreting the results of 
the net Impact analysis by program activity. That 1s, 1t will 
help 1n determining whether the estimated net Impacts by program 
activity are likely to truly reflect the relative effectiveness 
of different activities, or merely represent the fact that 
more-advantaged Individuals are assigned to certain activities, 
while less-advantaged Individuals are assigned to other 
activities . 

• The process analysis will also Identify major differences 1n the 
content of program activities across SOAs, as well as 
differences 1n the ways 1n which similar program activities are 
recorded 1n the JTPA MIS. This Information will be very useful 
Input to developing meaningful and consistent measures of 
program activities across SOAs. 

• Finally, the process analysis will be useful to the net Impact 
model by Identifying variables that should be Included 1n the 
model. For example, 1t may Identify SOA characteristics (e.g., 
service delivery strategies) that are quantifiable and that 
differ across SOAs so that they can be Included 1n the model to 
test whether the net impact of JTPA significantly differs across 
these dimensions. In addition, the process analysis may be able 
to identify SOA characteristics that could play the role of 
instruments in the net Impact analysis to correct for selection 
bias as discussed in Chapter 5. 

Thus, information from the process analysis will be valuable in 
conducting the net impact analysis and in Interpreting the findings. 

In addition to benefiting from the SOA process analysis, it should be 
noted that the net impact model may also produce Information that wou"H 
be of Interest to a process analysis . For example, the net impa- 1 
model may indicate that after adjusting for differences in participant 
characteristics and local labor market conditions that the net impact 
of JTPA is considerably larger in some SOAs than 1n others . The 
process analysis could then examine in more detail what it 1s about the 
specific SOAs that may account for such differences in net Impacts. If 
measures of specific SDA attributes that are potentially responsible 
for the different net impacts could be developed, these could be 
Included in subsequent net impact models to determine whether they 
account for the di f ferent net impact estimates across SOAs . Such 
ongoing Interaction between the process and net Impact models 
highlights the complementary nature of the two models and should result 
in an Improved understanding of the factors that affect program 
effectiveness. 

The net impact and gross impact model s are also quite complementary. 
Although both models are designed to address program~ef fecti veness 
questions , they di f fer in terms of the population subgroups of 
interest, the range of outcome measures of interest, and the types of 
comparisons being made. For example, as described in Chapter 2, the 
net Impact model is limited to adults only and, because it relies 




exclusively on administrative data sources, there are relatively few 
outcome measures that can be examined and only a limited number of 
personal characteristics can be Included 1n the model. On the other 
hand, the gross Impact model Includes youths as well as adults, an 
expanded set of labor market outcomes, and additional personal 
characteristics that can be Included 1n the model. As such, the gross 
Impact model can be used to answer certain effectiveness questions for 
youths served by JTPA and may also be able to provide Information on 
the mechanisms through which JTPA affects adults' earnings and welfare 
dependency. That Is, because of the additional outcomes that are 
available, a gross Impact analysis may be able to provide some evidence 
on whether the earnings changes are likely due to changes in wage 
rates, changes In hours worked per week, or changes In weeks worked per 
year, although a comparison group Is necessary to provide definitive 
evidence on these Issues. 

As the above discussion Indicates, because the focus of each of the 
models Is quite different, the Information generated Is very 
complementary. Taken together, the three models form a comprehensive 
analysis package that can provide much-needed Information on JTPA 
programs that will help states and SOAs meet their Increased 
accountability responsibilities under the Act. 

Finally, 1n addition to providing complementary Information on 
different subgroups and outcome measures, Information from the gross 
and net Impact models may shed light on Important methodological Issues 
that affect the validity of analyses of program Impacts In general, and 
that would be of considerable Interest to the research community. For 
example, the gross Impact model uses Interview-reported earnings, 
whereas the net Impact model relies on UI Wage Records to create 
measures of earnings. As discussed in the evaluation guide, there are 
advantages and disadvantages to both approaches. By Implementing the 
gross Impact model and by estimating gross Impacts using the net Impact 
design and with the same samples of participants and comparison group 
members, It would be possible to provide evidence on the extent to 
which the Impact results are sensitive to the use of the different 
sources of earnings data. In addition, because the gross Impact model 
has an expanded set of Independent variables available, by Implementing 
both models using the same analysis samples It would be possible to get 
some Idea of whether the net Impact estimates are sensitive to these 
omitted variables. Such comparisons would provide Important 
Information on the limitations of the different analyses and Indicate 
other Independent variables or outcome measures that would be Important 
to collect for subsequent program analysis. 
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